Need help with a weird kernel update panic.

[John Mellor]

I have one completely stock workstation F32 machine where kernel updates 
almost always cause a multiple-reboot panic problem.  This problem also 
occurred on F31, but not on releases before that. I'm stumped and need 
some help in figuring it out.

The symptoms vary in the number of reboots and the type of tertiary 
error, but are otherwise pretty similar.  It does not matter whether I 
use the Gnome update app or the CLI dnf method. After a number of 
reboots, the upgrade succeeds and Fedora behaves nortmally again.  I 
think that this only happens whenever the kernel is upgraded.

What I observe is that the machine is rebooted and on reboot, grub (I 
think) gets a halt for a 32-bit relocation error.  This sequence may 
happen twice.  Its an i7 with plenty of memory and an SSD boot disk, so 
the 32-bit thing is confusing.  To get around this error, I powercycle 
the box and get into the next stage of the problem.  One the 2nd or 3rd 
reboot, I usually see a halt with an access outside of the kernel space, 
although with the update this morning, I had a kernel panic instead.  
Cold-booting again, and the update is installed, and the last reboot and 
I'm up on the new updates.

After that, the machine behaves normally until the next kernel updates.  
I assume that there is some incorrectly-asynchronous operation in grub 
related to the update entry, but I can find no grub logs to dig into 
this problem.  I have several other machines that do not see this 
problem.  I dug around in the fedora bugs, but not knowing what to look 
for, I'm basically blind.  Its a pretty serious bug, especially if the 
machine is remote.  Does anyone have a way out of this?

--

John Mellor

[Roger Heflin]

What you are saying does not exactly match what I have previously
seen, but there is a known feature with using a journaling filesystem
(ext4-journal, or xfs) for /boot, if only the journal is updated and
if it is not yet replayed  into the non-journal then grub will not be
able to find the new files/updated files (grub filesystem code is
simple and does not process the journal so if critical updates are
still in the journal then those updates(changed file, new files)
cannot be seen).  To get this one generally has to do the update and
almost immediately reboot (within a few minutes though in some cases,
note syncing the does not replay the journal).   The fix is to boot up
with a kernel that it can still find and/or livecd and mount /boot so
that the journal gets replayed, or fsck boot so that the journal gets
replayed.

Long term the solution is to move boot to a non-journaled fs (ext
without a journal) or after each update umount/mount /boot(before
reboot)..  If /boot is not separated then you cannot umount/mount it
to get the journal to replay.  There is a second method to force a
journal replay, but reports say that one often "hangs" when /boot is
not separate so is not a reliable solution.    There were some
detailed posts on this several years ago with reliable commenters
confirming the behavior.  I have also personally seen the issue a
number of times and mount /boot and/or fscking corrects it (replays
journal).

On Fri, May 8, 2020 at 8:52 AM John Mellor <john.mellor(a)gmail.com&gt; wrote:
>
> I have one completely stock workstation F32 machine where kernel updates
> almost always cause a multiple-reboot panic problem.  This problem also
> occurred on F31, but not on releases before that. I'm stumped and need
> some help in figuring it out.
>
> The symptoms vary in the number of reboots and the type of tertiary
> error, but are otherwise pretty similar.  It does not matter whether I
> use the Gnome update app or the CLI dnf method. After a number of
> reboots, the upgrade succeeds and Fedora behaves nortmally again.  I
> think that this only happens whenever the kernel is upgraded.
>
> What I observe is that the machine is rebooted and on reboot, grub (I
> think) gets a halt for a 32-bit relocation error.  This sequence may
> happen twice.  Its an i7 with plenty of memory and an SSD boot disk, so
> the 32-bit thing is confusing.  To get around this error, I powercycle
> the box and get into the next stage of the problem.  One the 2nd or 3rd
> reboot, I usually see a halt with an access outside of the kernel space,
> although with the update this morning, I had a kernel panic instead.
> Cold-booting again, and the update is installed, and the last reboot and
> I'm up on the new updates.
>
> After that, the machine behaves normally until the next kernel updates.
> I assume that there is some incorrectly-asynchronous operation in grub
> related to the update entry, but I can find no grub logs to dig into
> this problem.  I have several other machines that do not see this
> problem.  I dug around in the fedora bugs, but not knowing what to look
> for, I'm basically blind.  Its a pretty serious bug, especially if the
> machine is remote.  Does anyone have a way out of this?
>
> --
>
> John Mellor
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[John Mellor]

Interesting!  This machine does reboot in about 5secs and the other 
machines take longer, so it makes sense.  My /boot is mounted just like 
/home and / as follows:

    /dev/sda1 on /boot type ext4 (rw,relatime,seclabel)

I assume that a symple sync would flush the journal.  Its pretty easy to 
do a sync;sync if updating using the CLI, but not possible when using 
the GUI.  Is this a Fedora bug where the journal is not correctly 
flushed on the reboot?  Should I modify that mount entry or do achattr 
change to workaround the bug?


On 2020-05-08 11:11 a.m., Roger Heflin wrote:
> What you are saying does not exactly match what I have previously
> seen, but there is a known feature with using a journaling filesystem
> (ext4-journal, or xfs) for /boot, if only the journal is updated and
> if it is not yet replayed  into the non-journal then grub will not be
> able to find the new files/updated files (grub filesystem code is
> simple and does not process the journal so if critical updates are
> still in the journal then those updates(changed file, new files)
> cannot be seen).  To get this one generally has to do the update and
> almost immediately reboot (within a few minutes though in some cases,
> note syncing the does not replay the journal).   The fix is to boot up
> with a kernel that it can still find and/or livecd and mount /boot so
> that the journal gets replayed, or fsck boot so that the journal gets
> replayed.
>
> Long term the solution is to move boot to a non-journaled fs (ext
> without a journal) or after each update umount/mount /boot(before
> reboot)..  If /boot is not separated then you cannot umount/mount it
> to get the journal to replay.  There is a second method to force a
> journal replay, but reports say that one often "hangs" when /boot is
> not separate so is not a reliable solution.    There were some
> detailed posts on this several years ago with reliable commenters
> confirming the behavior.  I have also personally seen the issue a
> number of times and mount /boot and/or fscking corrects it (replays
> journal).
>
> On Fri, May 8, 2020 at 8:52 AM John Mellor <john.mellor(a)gmail.com&gt; wrote:
>> I have one completely stock workstation F32 machine where kernel updates
>> almost always cause a multiple-reboot panic problem.  This problem also
>> occurred on F31, but not on releases before that. I'm stumped and need
>> some help in figuring it out.
>>
>> The symptoms vary in the number of reboots and the type of tertiary
>> error, but are otherwise pretty similar.  It does not matter whether I
>> use the Gnome update app or the CLI dnf method. After a number of
>> reboots, the upgrade succeeds and Fedora behaves nortmally again.  I
>> think that this only happens whenever the kernel is upgraded.
>>
>> What I observe is that the machine is rebooted and on reboot, grub (I
>> think) gets a halt for a 32-bit relocation error.  This sequence may
>> happen twice.  Its an i7 with plenty of memory and an SSD boot disk, so
>> the 32-bit thing is confusing.  To get around this error, I powercycle
>> the box and get into the next stage of the problem.  One the 2nd or 3rd
>> reboot, I usually see a halt with an access outside of the kernel space,
>> although with the update this morning, I had a kernel panic instead.
>> Cold-booting again, and the update is installed, and the last reboot and
>> I'm up on the new updates.
>>
>> After that, the machine behaves normally until the next kernel updates.
>> I assume that there is some incorrectly-asynchronous operation in grub
>> related to the update entry, but I can find no grub logs to dig into
>> this problem.  I have several other machines that do not see this
>> problem.  I dug around in the fedora bugs, but not knowing what to look
>> for, I'm basically blind.  Its a pretty serious bug, especially if the
>> machine is remote.  Does anyone have a way out of this?
>>
>> --
>>
>> John Mellor
>> _______________________________________________
>> users mailing list -- users(a)lists.fedoraproject.org
>> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Roger Heflin]

A sync will flush the writes to the journal were the data is safe.  It
will not force a replay of the journal.

Nothing except removing the journal from the ext4 filesystem will fix it.

This is not a fedora bug, this is a long standing
kernel/grub/filesystem interaction bug (all who use a journaled
filesystem have this bug).

See tune2fs and something like -O ^has_journal will turn off the
journal.  It has to be done unmounted and verify that your fstab entry
will remounted it.

Check /proc/mounts having data=XXX (probably ordered) says you have a
journal, after the umount+above tune2fs+remount the data=ordered will
be gone.

On Fri, May 8, 2020 at 10:53 AM John Mellor <john.mellor(a)gmail.com&gt; wrote:
>
> Interesting!  This machine does reboot in about 5secs and the other
> machines take longer, so it makes sense.  My /boot is mounted just like
> /home and / as follows:
>
>     /dev/sda1 on /boot type ext4 (rw,relatime,seclabel)
>
> I assume that a symple sync would flush the journal.  Its pretty easy to
> do a sync;sync if updating using the CLI, but not possible when using
> the GUI.  Is this a Fedora bug where the journal is not correctly
> flushed on the reboot?  Should I modify that mount entry or do achattr
> change to workaround the bug?
>
>
> On 2020-05-08 11:11 a.m., Roger Heflin wrote:
> > What you are saying does not exactly match what I have previously
> > seen, but there is a known feature with using a journaling filesystem
> > (ext4-journal, or xfs) for /boot, if only the journal is updated and
> > if it is not yet replayed  into the non-journal then grub will not be
> > able to find the new files/updated files (grub filesystem code is
> > simple and does not process the journal so if critical updates are
> > still in the journal then those updates(changed file, new files)
> > cannot be seen).  To get this one generally has to do the update and
> > almost immediately reboot (within a few minutes though in some cases,
> > note syncing the does not replay the journal).   The fix is to boot up
> > with a kernel that it can still find and/or livecd and mount /boot so
> > that the journal gets replayed, or fsck boot so that the journal gets
> > replayed.
> >
> > Long term the solution is to move boot to a non-journaled fs (ext
> > without a journal) or after each update umount/mount /boot(before
> > reboot)..  If /boot is not separated then you cannot umount/mount it
> > to get the journal to replay.  There is a second method to force a
> > journal replay, but reports say that one often "hangs" when /boot is
> > not separate so is not a reliable solution.    There were some
> > detailed posts on this several years ago with reliable commenters
> > confirming the behavior.  I have also personally seen the issue a
> > number of times and mount /boot and/or fscking corrects it (replays
> > journal).
> >
> > On Fri, May 8, 2020 at 8:52 AM John Mellor <john.mellor(a)gmail.com&gt; wrote:
> >> I have one completely stock workstation F32 machine where kernel updates
> >> almost always cause a multiple-reboot panic problem.  This problem also
> >> occurred on F31, but not on releases before that. I'm stumped and need
> >> some help in figuring it out.
> >>
> >> The symptoms vary in the number of reboots and the type of tertiary
> >> error, but are otherwise pretty similar.  It does not matter whether I
> >> use the Gnome update app or the CLI dnf method. After a number of
> >> reboots, the upgrade succeeds and Fedora behaves nortmally again.  I
> >> think that this only happens whenever the kernel is upgraded.
> >>
> >> What I observe is that the machine is rebooted and on reboot, grub (I
> >> think) gets a halt for a 32-bit relocation error.  This sequence may
> >> happen twice.  Its an i7 with plenty of memory and an SSD boot disk, so
> >> the 32-bit thing is confusing.  To get around this error, I powercycle
> >> the box and get into the next stage of the problem.  One the 2nd or 3rd
> >> reboot, I usually see a halt with an access outside of the kernel space,
> >> although with the update this morning, I had a kernel panic instead.
> >> Cold-booting again, and the update is installed, and the last reboot and
> >> I'm up on the new updates.
> >>
> >> After that, the machine behaves normally until the next kernel updates.
> >> I assume that there is some incorrectly-asynchronous operation in grub
> >> related to the update entry, but I can find no grub logs to dig into
> >> this problem.  I have several other machines that do not see this
> >> problem.  I dug around in the fedora bugs, but not knowing what to look
> >> for, I'm basically blind.  Its a pretty serious bug, especially if the
> >> machine is remote.  Does anyone have a way out of this?
> >>
> >> --
> >>
> >> John Mellor
> >> _______________________________________________
> >> users mailing list -- users(a)lists.fedoraproject.org
> >> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> >> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> > _______________________________________________
> > users mailing list -- users(a)lists.fedoraproject.org
> > To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Mauricio Tavares]

On Fri, May 8, 2020 at 12:12 PM Roger Heflin <rogerheflin(a)gmail.com&gt; wrote:
>  A sync will flush the writes to the journal were the data is safe.  It
>  will not force a replay of the journal.
>
>  Nothing except removing the journal from the ext4 filesystem will fix it.
>
>  This is not a fedora bug, this is a long standing
>  kernel/grub/filesystem interaction bug (all who use a journaled
>  filesystem have this bug).
>
>  See tune2fs and something like -O ^has_journal will turn off the
>  journal.  It has to be done unmounted and verify that your fstab entry
>  will remounted it.
>
>  Check /proc/mounts having data=XXX (probably ordered) says you have a
>  journal, after the umount+above tune2fs+remount the data=ordered will
>  be gone.
>  
      Interesting. I have a box which have been running for years with

/dev/sdb1 /boot ext4
rw,seclabel,noatime,barrier=1,stripe=32,data=ordered,discard 0 0

and so far never borked on me.

> On Fri, May 8, 2020 at 10:53 AM John Mellor <john.mellor(a)gmail.com&gt; wrote:
> >
> > Interesting!  This machine does reboot in about 5secs and the other
> > machines take longer, so it makes sense.  My /boot is mounted just like
> > /home and / as follows:
> >
> >     /dev/sda1 on /boot type ext4 (rw,relatime,seclabel)
> >
> > I assume that a symple sync would flush the journal.  Its pretty easy to
> > do a sync;sync if updating using the CLI, but not possible when using
> > the GUI.  Is this a Fedora bug where the journal is not correctly
> > flushed on the reboot?  Should I modify that mount entry or do achattr
> > change to workaround the bug?
> >
> >
> > On 2020-05-08 11:11 a.m., Roger Heflin wrote:
> > > What you are saying does not exactly match what I have previously
> > > seen, but there is a known feature with using a journaling filesystem
> > > (ext4-journal, or xfs) for /boot, if only the journal is updated and
> > > if it is not yet replayed  into the non-journal then grub will not be
> > > able to find the new files/updated files (grub filesystem code is
> > > simple and does not process the journal so if critical updates are
> > > still in the journal then those updates(changed file, new files)
> > > cannot be seen).  To get this one generally has to do the update and
> > > almost immediately reboot (within a few minutes though in some cases,
> > > note syncing the does not replay the journal).   The fix is to boot up
> > > with a kernel that it can still find and/or livecd and mount /boot so
> > > that the journal gets replayed, or fsck boot so that the journal gets
> > > replayed.
> > >
> > > Long term the solution is to move boot to a non-journaled fs (ext
> > > without a journal) or after each update umount/mount /boot(before
> > > reboot)..  If /boot is not separated then you cannot umount/mount it
> > > to get the journal to replay.  There is a second method to force a
> > > journal replay, but reports say that one often "hangs" when /boot
is
> > > not separate so is not a reliable solution.    There were some
> > > detailed posts on this several years ago with reliable commenters
> > > confirming the behavior.  I have also personally seen the issue a
> > > number of times and mount /boot and/or fscking corrects it (replays
> > > journal).
> > >
> > > On Fri, May 8, 2020 at 8:52 AM John Mellor <john.mellor(a)gmail.com&gt;
wrote:
> > >> I have one completely stock workstation F32 machine where kernel
updates
> > >> almost always cause a multiple-reboot panic problem.  This problem
also
> > >> occurred on F31, but not on releases before that. I'm stumped and
need
> > >> some help in figuring it out.
> > >>
> > >> The symptoms vary in the number of reboots and the type of tertiary
> > >> error, but are otherwise pretty similar.  It does not matter whether I
> > >> use the Gnome update app or the CLI dnf method. After a number of
> > >> reboots, the upgrade succeeds and Fedora behaves nortmally again.  I
> > >> think that this only happens whenever the kernel is upgraded.
> > >>
> > >> What I observe is that the machine is rebooted and on reboot, grub (I
> > >> think) gets a halt for a 32-bit relocation error.  This sequence may
> > >> happen twice.  Its an i7 with plenty of memory and an SSD boot disk,
so
> > >> the 32-bit thing is confusing.  To get around this error, I powercycle
> > >> the box and get into the next stage of the problem.  One the 2nd or
3rd
> > >> reboot, I usually see a halt with an access outside of the kernel
space,
> > >> although with the update this morning, I had a kernel panic instead.
> > >> Cold-booting again, and the update is installed, and the last reboot
and
> > >> I'm up on the new updates.
> > >>
> > >> After that, the machine behaves normally until the next kernel
updates.
> > >> I assume that there is some incorrectly-asynchronous operation in grub
> > >> related to the update entry, but I can find no grub logs to dig into
> > >> this problem.  I have several other machines that do not see this
> > >> problem.  I dug around in the fedora bugs, but not knowing what to
look
> > >> for, I'm basically blind.  Its a pretty serious bug, especially if
the
> > >> machine is remote.  Does anyone have a way out of this?
> > >>
> > >> --
> > >>
> > >> John Mellor
> > >> _______________________________________________
> > >> users mailing list -- users(a)lists.fedoraproject.org
> > >> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> > >> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > >> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> > >> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> > > _______________________________________________
> > > users mailing list -- users(a)lists.fedoraproject.org
> > > To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> > > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> > _______________________________________________
> > users mailing list -- users(a)lists.fedoraproject.org
> > To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Roger Heflin]

You have to hit the timing right.  ie install the kernel package and
as quickly as possible reboot (automated, or very efficient).

And if the update is more than just kernel, that may slowdown the
process enough that the immediate reboot won't be quick enough.

I have seen it 3-5 times and that is over a huge number of machines,
in those the machines were booted and failed multiple times before
someone livecd booted it ran fsck'ed and/or mounted /boot, and it
found the files after that.

On Fri, May 8, 2020 at 2:00 PM Mauricio Tavares <raubvogel(a)gmail.com&gt; wrote:
>
> On Fri, May 8, 2020 at 12:12 PM Roger Heflin <rogerheflin(a)gmail.com&gt; wrote:
> >
> > A sync will flush the writes to the journal were the data is safe.  It
> > will not force a replay of the journal.
> >
> > Nothing except removing the journal from the ext4 filesystem will fix it.
> >
> > This is not a fedora bug, this is a long standing
> > kernel/grub/filesystem interaction bug (all who use a journaled
> > filesystem have this bug).
> >
> > See tune2fs and something like -O ^has_journal will turn off the
> > journal.  It has to be done unmounted and verify that your fstab entry
> > will remounted it.
> >
> > Check /proc/mounts having data=XXX (probably ordered) says you have a
> > journal, after the umount+above tune2fs+remount the data=ordered will
> > be gone.
> >
>       Interesting. I have a box which have been running for years with
>
> /dev/sdb1 /boot ext4
> rw,seclabel,noatime,barrier=1,stripe=32,data=ordered,discard 0 0
>
> and so far never borked on me.
>
> > On Fri, May 8, 2020 at 10:53 AM John Mellor <john.mellor(a)gmail.com&gt;
wrote:
> > >
> > > Interesting!  This machine does reboot in about 5secs and the other
> > > machines take longer, so it makes sense.  My /boot is mounted just like
> > > /home and / as follows:
> > >
> > >     /dev/sda1 on /boot type ext4 (rw,relatime,seclabel)
> > >
> > > I assume that a symple sync would flush the journal.  Its pretty easy to
> > > do a sync;sync if updating using the CLI, but not possible when using
> > > the GUI.  Is this a Fedora bug where the journal is not correctly
> > > flushed on the reboot?  Should I modify that mount entry or do achattr
> > > change to workaround the bug?
> > >
> > >
> > > On 2020-05-08 11:11 a.m., Roger Heflin wrote:
> > > > What you are saying does not exactly match what I have previously
> > > > seen, but there is a known feature with using a journaling filesystem
> > > > (ext4-journal, or xfs) for /boot, if only the journal is updated and
> > > > if it is not yet replayed  into the non-journal then grub will not be
> > > > able to find the new files/updated files (grub filesystem code is
> > > > simple and does not process the journal so if critical updates are
> > > > still in the journal then those updates(changed file, new files)
> > > > cannot be seen).  To get this one generally has to do the update and
> > > > almost immediately reboot (within a few minutes though in some cases,
> > > > note syncing the does not replay the journal).   The fix is to boot
up
> > > > with a kernel that it can still find and/or livecd and mount /boot so
> > > > that the journal gets replayed, or fsck boot so that the journal gets
> > > > replayed.
> > > >
> > > > Long term the solution is to move boot to a non-journaled fs (ext
> > > > without a journal) or after each update umount/mount /boot(before
> > > > reboot)..  If /boot is not separated then you cannot umount/mount it
> > > > to get the journal to replay.  There is a second method to force a
> > > > journal replay, but reports say that one often "hangs" when
/boot is
> > > > not separate so is not a reliable solution.    There were some
> > > > detailed posts on this several years ago with reliable commenters
> > > > confirming the behavior.  I have also personally seen the issue a
> > > > number of times and mount /boot and/or fscking corrects it (replays
> > > > journal).
> > > >
> > > > On Fri, May 8, 2020 at 8:52 AM John Mellor
<john.mellor(a)gmail.com&gt; wrote:
> > > >> I have one completely stock workstation F32 machine where kernel
updates
> > > >> almost always cause a multiple-reboot panic problem.  This problem
also
> > > >> occurred on F31, but not on releases before that. I'm stumped
and need
> > > >> some help in figuring it out.
> > > >>
> > > >> The symptoms vary in the number of reboots and the type of
tertiary
> > > >> error, but are otherwise pretty similar.  It does not matter
whether I
> > > >> use the Gnome update app or the CLI dnf method. After a number of
> > > >> reboots, the upgrade succeeds and Fedora behaves nortmally again. 
I
> > > >> think that this only happens whenever the kernel is upgraded.
> > > >>
> > > >> What I observe is that the machine is rebooted and on reboot, grub
(I
> > > >> think) gets a halt for a 32-bit relocation error.  This sequence
may
> > > >> happen twice.  Its an i7 with plenty of memory and an SSD boot
disk, so
> > > >> the 32-bit thing is confusing.  To get around this error, I
powercycle
> > > >> the box and get into the next stage of the problem.  One the 2nd
or 3rd
> > > >> reboot, I usually see a halt with an access outside of the kernel
space,
> > > >> although with the update this morning, I had a kernel panic
instead.
> > > >> Cold-booting again, and the update is installed, and the last
reboot and
> > > >> I'm up on the new updates.
> > > >>
> > > >> After that, the machine behaves normally until the next kernel
updates.
> > > >> I assume that there is some incorrectly-asynchronous operation in
grub
> > > >> related to the update entry, but I can find no grub logs to dig
into
> > > >> this problem.  I have several other machines that do not see this
> > > >> problem.  I dug around in the fedora bugs, but not knowing what to
look
> > > >> for, I'm basically blind.  Its a pretty serious bug,
especially if the
> > > >> machine is remote.  Does anyone have a way out of this?
> > > >>
> > > >> --
> > > >>
> > > >> John Mellor
> > > >> _______________________________________________
> > > >> users mailing list -- users(a)lists.fedoraproject.org
> > > >> To unsubscribe send an email to
users-leave(a)lists.fedoraproject.org
> > > >> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > >> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > >> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> > > > _______________________________________________
> > > > users mailing list -- users(a)lists.fedoraproject.org
> > > > To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> > > > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> > > _______________________________________________
> > > users mailing list -- users(a)lists.fedoraproject.org
> > > To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> > > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> > _______________________________________________
> > users mailing list -- users(a)lists.fedoraproject.org
> > To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[John Mellor]

I like the idea of just removing the journal tag, but I don't think that 
I can modify the /boot filesystem.  Doing a umount works, but tune2fs 
claims that an e2fsck is required.  Running e2fsck says that the 
filesystem is still mounted, even though it is not.  Doing a fuser 
/dev/sda1 shows a large number of /proc/fd entries using it, even though 
it successfully umounted.

So, something is referencing the filesystem in a very bad way, and not 
as a mounted fs.

So, I'm still stuck.  How do you successfully modify /boot to not be 
journalled?


On 2020-05-08 3:25 p.m., Roger Heflin wrote:
> You have to hit the timing right.  ie install the kernel package and
> as quickly as possible reboot (automated, or very efficient).
>
> And if the update is more than just kernel, that may slowdown the
> process enough that the immediate reboot won't be quick enough.
>
> I have seen it 3-5 times and that is over a huge number of machines,
> in those the machines were booted and failed multiple times before
> someone livecd booted it ran fsck'ed and/or mounted /boot, and it
> found the files after that.
>
> On Fri, May 8, 2020 at 2:00 PM Mauricio Tavares <raubvogel(a)gmail.com&gt; wrote:
>> On Fri, May 8, 2020 at 12:12 PM Roger Heflin <rogerheflin(a)gmail.com&gt;
wrote:
>>> A sync will flush the writes to the journal were the data is safe.  It
>>> will not force a replay of the journal.
>>>
>>> Nothing except removing the journal from the ext4 filesystem will fix it.
>>>
>>> This is not a fedora bug, this is a long standing
>>> kernel/grub/filesystem interaction bug (all who use a journaled
>>> filesystem have this bug).
>>>
>>> See tune2fs and something like -O ^has_journal will turn off the
>>> journal.  It has to be done unmounted and verify that your fstab entry
>>> will remounted it.
>>>
>>> Check /proc/mounts having data=XXX (probably ordered) says you have a
>>> journal, after the umount+above tune2fs+remount the data=ordered will
>>> be gone.
>>>
>>        Interesting. I have a box which have been running for years with
>>
>> /dev/sdb1 /boot ext4
>> rw,seclabel,noatime,barrier=1,stripe=32,data=ordered,discard 0 0
>>
>> and so far never borked on me.
>>
>>> On Fri, May 8, 2020 at 10:53 AM John Mellor <john.mellor(a)gmail.com&gt;
wrote:
>>>> Interesting!  This machine does reboot in about 5secs and the other
>>>> machines take longer, so it makes sense.  My /boot is mounted just like
>>>> /home and / as follows:
>>>>
>>>>      /dev/sda1 on /boot type ext4 (rw,relatime,seclabel)
>>>>
>>>> I assume that a symple sync would flush the journal.  Its pretty easy to
>>>> do a sync;sync if updating using the CLI, but not possible when using
>>>> the GUI.  Is this a Fedora bug where the journal is not correctly
>>>> flushed on the reboot?  Should I modify that mount entry or do achattr
>>>> change to workaround the bug?
>>>>
>>>>
>>>> On 2020-05-08 11:11 a.m., Roger Heflin wrote:
>>>>> What you are saying does not exactly match what I have previously
>>>>> seen, but there is a known feature with using a journaling
filesystem
>>>>> (ext4-journal, or xfs) for /boot, if only the journal is updated and
>>>>> if it is not yet replayed  into the non-journal then grub will not
be
>>>>> able to find the new files/updated files (grub filesystem code is
>>>>> simple and does not process the journal so if critical updates are
>>>>> still in the journal then those updates(changed file, new files)
>>>>> cannot be seen).  To get this one generally has to do the update and
>>>>> almost immediately reboot (within a few minutes though in some
cases,
>>>>> note syncing the does not replay the journal).   The fix is to boot
up
>>>>> with a kernel that it can still find and/or livecd and mount /boot
so
>>>>> that the journal gets replayed, or fsck boot so that the journal
gets
>>>>> replayed.
>>>>>
>>>>> Long term the solution is to move boot to a non-journaled fs (ext
>>>>> without a journal) or after each update umount/mount /boot(before
>>>>> reboot)..  If /boot is not separated then you cannot umount/mount it
>>>>> to get the journal to replay.  There is a second method to force a
>>>>> journal replay, but reports say that one often "hangs" when
/boot is
>>>>> not separate so is not a reliable solution.    There were some
>>>>> detailed posts on this several years ago with reliable commenters
>>>>> confirming the behavior.  I have also personally seen the issue a
>>>>> number of times and mount /boot and/or fscking corrects it (replays
>>>>> journal).
>>>>>
>>>>> On Fri, May 8, 2020 at 8:52 AM John Mellor
<john.mellor(a)gmail.com&gt; wrote:
>>>>>> I have one completely stock workstation F32 machine where kernel
updates
>>>>>> almost always cause a multiple-reboot panic problem.  This
problem also
>>>>>> occurred on F31, but not on releases before that. I'm stumped
and need
>>>>>> some help in figuring it out.
>>>>>>
>>>>>> The symptoms vary in the number of reboots and the type of
tertiary
>>>>>> error, but are otherwise pretty similar.  It does not matter
whether I
>>>>>> use the Gnome update app or the CLI dnf method. After a number
of
>>>>>> reboots, the upgrade succeeds and Fedora behaves nortmally again.
 I
>>>>>> think that this only happens whenever the kernel is upgraded.
>>>>>>
>>>>>> What I observe is that the machine is rebooted and on reboot,
grub (I
>>>>>> think) gets a halt for a 32-bit relocation error.  This sequence
may
>>>>>> happen twice.  Its an i7 with plenty of memory and an SSD boot
disk, so
>>>>>> the 32-bit thing is confusing.  To get around this error, I
powercycle
>>>>>> the box and get into the next stage of the problem.  One the 2nd
or 3rd
>>>>>> reboot, I usually see a halt with an access outside of the kernel
space,
>>>>>> although with the update this morning, I had a kernel panic
instead.
>>>>>> Cold-booting again, and the update is installed, and the last
reboot and
>>>>>> I'm up on the new updates.
>>>>>>
>>>>>> After that, the machine behaves normally until the next kernel
updates.
>>>>>> I assume that there is some incorrectly-asynchronous operation in
grub
>>>>>> related to the update entry, but I can find no grub logs to dig
into
>>>>>> this problem.  I have several other machines that do not see
this
>>>>>> problem.  I dug around in the fedora bugs, but not knowing what
to look
>>>>>> for, I'm basically blind.  Its a pretty serious bug,
especially if the
>>>>>> machine is remote.  Does anyone have a way out of this?
>>>>>>
>>>>>> --
>>>>>>
>>>>>> John Mellor
>>>>>> _______________________________________________
>

[Roger Heflin]

Did you do a lazy unmount?  Any other umount will fail if there are
open files, and lazy does not actually unmount it until whatever has
the files open closes them, which generally means it never gets
unmounted.


On Fri, May 8, 2020 at 5:53 PM John Mellor <john.mellor(a)gmail.com&gt; wrote:
>
> I like the idea of just removing the journal tag, but I don't think that
> I can modify the /boot filesystem.  Doing a umount works, but tune2fs
> claims that an e2fsck is required.  Running e2fsck says that the
> filesystem is still mounted, even though it is not.  Doing a fuser
> /dev/sda1 shows a large number of /proc/fd entries using it, even though
> it successfully umounted.
>
> So, something is referencing the filesystem in a very bad way, and not
> as a mounted fs.
>
> So, I'm still stuck.  How do you successfully modify /boot to not be
> journalled?
>
>
> On 2020-05-08 3:25 p.m., Roger Heflin wrote:
> > You have to hit the timing right.  ie install the kernel package and
> > as quickly as possible reboot (automated, or very efficient).
> >
> > And if the update is more than just kernel, that may slowdown the
> > process enough that the immediate reboot won't be quick enough.
> >
> > I have seen it 3-5 times and that is over a huge number of machines,
> > in those the machines were booted and failed multiple times before
> > someone livecd booted it ran fsck'ed and/or mounted /boot, and it
> > found the files after that.
> >
> > On Fri, May 8, 2020 at 2:00 PM Mauricio Tavares <raubvogel(a)gmail.com&gt;
wrote:
> >> On Fri, May 8, 2020 at 12:12 PM Roger Heflin <rogerheflin(a)gmail.com&gt;
wrote:
> >>> A sync will flush the writes to the journal were the data is safe.  It
> >>> will not force a replay of the journal.
> >>>
> >>> Nothing except removing the journal from the ext4 filesystem will fix
it.
> >>>
> >>> This is not a fedora bug, this is a long standing
> >>> kernel/grub/filesystem interaction bug (all who use a journaled
> >>> filesystem have this bug).
> >>>
> >>> See tune2fs and something like -O ^has_journal will turn off the
> >>> journal.  It has to be done unmounted and verify that your fstab entry
> >>> will remounted it.
> >>>
> >>> Check /proc/mounts having data=XXX (probably ordered) says you have a
> >>> journal, after the umount+above tune2fs+remount the data=ordered will
> >>> be gone.
> >>>
> >>        Interesting. I have a box which have been running for years with
> >>
> >> /dev/sdb1 /boot ext4
> >> rw,seclabel,noatime,barrier=1,stripe=32,data=ordered,discard 0 0
> >>
> >> and so far never borked on me.
> >>
> >>> On Fri, May 8, 2020 at 10:53 AM John Mellor
<john.mellor(a)gmail.com&gt; wrote:
> >>>> Interesting!  This machine does reboot in about 5secs and the other
> >>>> machines take longer, so it makes sense.  My /boot is mounted just
like
> >>>> /home and / as follows:
> >>>>
> >>>>      /dev/sda1 on /boot type ext4 (rw,relatime,seclabel)
> >>>>
> >>>> I assume that a symple sync would flush the journal.  Its pretty
easy to
> >>>> do a sync;sync if updating using the CLI, but not possible when
using
> >>>> the GUI.  Is this a Fedora bug where the journal is not correctly
> >>>> flushed on the reboot?  Should I modify that mount entry or do
achattr
> >>>> change to workaround the bug?
> >>>>
> >>>>
> >>>> On 2020-05-08 11:11 a.m., Roger Heflin wrote:
> >>>>> What you are saying does not exactly match what I have
previously
> >>>>> seen, but there is a known feature with using a journaling
filesystem
> >>>>> (ext4-journal, or xfs) for /boot, if only the journal is updated
and
> >>>>> if it is not yet replayed  into the non-journal then grub will
not be
> >>>>> able to find the new files/updated files (grub filesystem code
is
> >>>>> simple and does not process the journal so if critical updates
are
> >>>>> still in the journal then those updates(changed file, new
files)
> >>>>> cannot be seen).  To get this one generally has to do the update
and
> >>>>> almost immediately reboot (within a few minutes though in some
cases,
> >>>>> note syncing the does not replay the journal).   The fix is to
boot up
> >>>>> with a kernel that it can still find and/or livecd and mount
/boot so
> >>>>> that the journal gets replayed, or fsck boot so that the journal
gets
> >>>>> replayed.
> >>>>>
> >>>>> Long term the solution is to move boot to a non-journaled fs
(ext
> >>>>> without a journal) or after each update umount/mount
/boot(before
> >>>>> reboot)..  If /boot is not separated then you cannot
umount/mount it
> >>>>> to get the journal to replay.  There is a second method to force
a
> >>>>> journal replay, but reports say that one often "hangs"
when /boot is
> >>>>> not separate so is not a reliable solution.    There were some
> >>>>> detailed posts on this several years ago with reliable
commenters
> >>>>> confirming the behavior.  I have also personally seen the issue
a
> >>>>> number of times and mount /boot and/or fscking corrects it
(replays
> >>>>> journal).
> >>>>>
> >>>>> On Fri, May 8, 2020 at 8:52 AM John Mellor
<john.mellor(a)gmail.com&gt; wrote:
> >>>>>> I have one completely stock workstation F32 machine where
kernel updates
> >>>>>> almost always cause a multiple-reboot panic problem.  This
problem also
> >>>>>> occurred on F31, but not on releases before that. I'm
stumped and need
> >>>>>> some help in figuring it out.
> >>>>>>
> >>>>>> The symptoms vary in the number of reboots and the type of
tertiary
> >>>>>> error, but are otherwise pretty similar.  It does not matter
whether I
> >>>>>> use the Gnome update app or the CLI dnf method. After a
number of
> >>>>>> reboots, the upgrade succeeds and Fedora behaves nortmally
again.  I
> >>>>>> think that this only happens whenever the kernel is
upgraded.
> >>>>>>
> >>>>>> What I observe is that the machine is rebooted and on
reboot, grub (I
> >>>>>> think) gets a halt for a 32-bit relocation error.  This
sequence may
> >>>>>> happen twice.  Its an i7 with plenty of memory and an SSD
boot disk, so
> >>>>>> the 32-bit thing is confusing.  To get around this error, I
powercycle
> >>>>>> the box and get into the next stage of the problem.  One the
2nd or 3rd
> >>>>>> reboot, I usually see a halt with an access outside of the
kernel space,
> >>>>>> although with the update this morning, I had a kernel panic
instead.
> >>>>>> Cold-booting again, and the update is installed, and the
last reboot and
> >>>>>> I'm up on the new updates.
> >>>>>>
> >>>>>> After that, the machine behaves normally until the next
kernel updates.
> >>>>>> I assume that there is some incorrectly-asynchronous
operation in grub
> >>>>>> related to the update entry, but I can find no grub logs to
dig into
> >>>>>> this problem.  I have several other machines that do not see
this
> >>>>>> problem.  I dug around in the fedora bugs, but not knowing
what to look
> >>>>>> for, I'm basically blind.  Its a pretty serious bug,
especially if the
> >>>>>> machine is remote.  Does anyone have a way out of this?
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> John Mellor
> >>>>>> _______________________________________________
> >
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[John Mellor]

Ok, I think I've finally managed to remove the journal on /boot. I did 
it by rebooting into an old F30 boot media in troubleshooting mode using 
'tune2fs -O ^has_journal'.  I was unable to do this on the disk even in 
single user mode, as it always marked the /boot partition as unclean, 
and always had /proc/fd/* files open.  This machine was upgraded all the 
way back from F23 to F32, so this odd behaviour may be an artifact of 
this history.

Now I need to wait for the next kernel upgrade, and see if that change 
fixes my crash/panic issue.

Waiting and watching.  Thanks for the pointer!


On 2020-05-08 12:11 p.m., Roger Heflin wrote:
> A sync will flush the writes to the journal were the data is safe.  It
> will not force a replay of the journal.
>
> Nothing except removing the journal from the ext4 filesystem will fix it.
>
> This is not a fedora bug, this is a long standing
> kernel/grub/filesystem interaction bug (all who use a journaled
> filesystem have this bug).
>
> See tune2fs and something like -O ^has_journal will turn off the
> journal.  It has to be done unmounted and verify that your fstab entry
> will remounted it.
>
> Check /proc/mounts having data=XXX (probably ordered) says you have a
> journal, after the umount+above tune2fs+remount the data=ordered will
> be gone.
>
> On Fri, May 8, 2020 at 10:53 AM John Mellor <john.mellor(a)gmail.com&gt; wrote:
>> Interesting!  This machine does reboot in about 5secs and the other
>> machines take longer, so it makes sense.  My /boot is mounted just like
>> /home and / as follows:
>>
>>      /dev/sda1 on /boot type ext4 (rw,relatime,seclabel)
>>
>> I assume that a symple sync would flush the journal.  Its pretty easy to
>> do a sync;sync if updating using the CLI, but not possible when using
>> the GUI.  Is this a Fedora bug where the journal is not correctly
>> flushed on the reboot?  Should I modify that mount entry or do achattr
>> change to workaround the bug?
>>
>>
>> On 2020-05-08 11:11 a.m., Roger Heflin wrote:
>>> What you are saying does not exactly match what I have previously
>>> seen, but there is a known feature with using a journaling filesystem
>>> (ext4-journal, or xfs) for /boot, if only the journal is updated and
>>> if it is not yet replayed  into the non-journal then grub will not be
>>> able to find the new files/updated files (grub filesystem code is
>>> simple and does not process the journal so if critical updates are
>>> still in the journal then those updates(changed file, new files)
>>> cannot be seen).  To get this one generally has to do the update and
>>> almost immediately reboot (within a few minutes though in some cases,
>>> note syncing the does not replay the journal).   The fix is to boot up
>>> with a kernel that it can still find and/or livecd and mount /boot so
>>> that the journal gets replayed, or fsck boot so that the journal gets
>>> replayed.
>>>
>>> Long term the solution is to move boot to a non-journaled fs (ext
>>> without a journal) or after each update umount/mount /boot(before
>>> reboot)..  If /boot is not separated then you cannot umount/mount it
>>> to get the journal to replay.  There is a second method to force a
>>> journal replay, but reports say that one often "hangs" when /boot
is
>>> not separate so is not a reliable solution.    There were some
>>> detailed posts on this several years ago with reliable commenters
>>> confirming the behavior.  I have also personally seen the issue a
>>> number of times and mount /boot and/or fscking corrects it (replays
>>> journal).
>>>
>>> On Fri, May 8, 2020 at 8:52 AM John Mellor <john.mellor(a)gmail.com&gt;
wrote:
>>>> I have one completely stock workstation F32 machine where kernel updates
>>>> almost always cause a multiple-reboot panic problem.  This problem also
>>>> occurred on F31, but not on releases before that. I'm stumped and
need
>>>> some help in figuring it out.
>>>>
>>>> The symptoms vary in the number of reboots and the type of tertiary
>>>> error, but are otherwise pretty similar.  It does not matter whether I
>>>> use the Gnome update app or the CLI dnf method. After a number of
>>>> reboots, the upgrade succeeds and Fedora behaves nortmally again.  I
>>>> think that this only happens whenever the kernel is upgraded.
>>>>
>>>> What I observe is that the machine is rebooted and on reboot, grub (I
>>>> think) gets a halt for a 32-bit relocation error.  This sequence may
>>>> happen twice.  Its an i7 with plenty of memory and an SSD boot disk, so
>>>> the 32-bit thing is confusing.  To get around this error, I powercycle
>>>> the box and get into the next stage of the problem.  One the 2nd or 3rd
>>>> reboot, I usually see a halt with an access outside of the kernel space,
>>>> although with the update this morning, I had a kernel panic instead.
>>>> Cold-booting again, and the update is installed, and the last reboot and
>>>> I'm up on the new updates.
>>>>
>>>> After that, the machine behaves normally until the next kernel updates.
>>>> I assume that there is some incorrectly-asynchronous operation in grub
>>>> related to the update entry, but I can find no grub logs to dig into
>>>> this problem.  I have several other machines that do not see this
>>>> problem.  I dug around in the fedora bugs, but not knowing what to look
>>>> for, I'm basically blind.  Its a pretty serious bug, especially if
the
>>>> machine is remote.  Does anyone have a way out of this?
>>>>
>>>> --
>>>>
>>>> John Mellor
>

[Ed Greshko]

On 2020-05-09 21:01, John Mellor wrote:
>  Now I need to wait for the next kernel upgrade, and see if that
> change fixes my crash/panic issue.  
Of course you could boot to the previous kernel, remove the latest one, reboot again to
the previous
kernel and allow the upgrade to test.

-- 
The key to getting good answers is to ask good questions.

[John Mellor]

On 2020-05-09 5:53 p.m., Ed Greshko wrote:
>  On 2020-05-09 21:01, John Mellor wrote:
> > Now I need to wait for the next kernel upgrade, and see if that change
> fixes my crash/panic issue.
>  Of course you could boot to the previous kernel, remove the latest one, reboot again to
> the previous
>  kernel and allow the upgrade to test. 
Many thanks, I think it worked.  The kernel upgrade this morning 
installed cleanly.

This is probably a bug in the upgrade process, in that there is no check 
or code to remove the journal on /boot that was put there by an earlier 
fresh install.

--

John Mellor

[Samuel Sieb]

On 5/12/20 5:15 AM, John Mellor wrote:
>  On 2020-05-09 5:53 p.m., Ed Greshko wrote:
> > On 2020-05-09 21:01, John Mellor wrote:
> >> Now I need to wait for the next kernel upgrade, and see if that 
> >> change fixes my crash/panic issue.
> > Of course you could boot to the previous kernel, remove the latest 
> > one, reboot again to the previous
> > kernel and allow the upgrade to test.
>  
>  Many thanks, I think it worked.  The kernel upgrade this morning 
>  installed cleanly.
>  
>  This is probably a bug in the upgrade process, in that there is no check 
>  or code to remove the journal on /boot that was put there by an earlier 
>  fresh install. 
It's not a bug.  There is no reason to remove the journal.  And for 
anyone not using LVM, most will have /boot as part of / and removing the 
journal from that would be a disaster.

[John Mellor]

Hi Samuel,
But that's the point.  A default install uses LVM, and therefore constructs
a separate /boot partition running EXT4 with a journal.  You may of course
decide to do your own partitions, but that setup is highly unusual and
rare.  Most people will have the default partitioning, and therefore have
the kernel upgrade bug.
--
John Mellor

On Tue, May 12, 2020 at 1:41 PM Samuel Sieb <samuel(a)sieb.net&gt; wrote:

>  On 5/12/20 5:15 AM, John Mellor wrote:
>  > On 2020-05-09 5:53 p.m., Ed Greshko wrote:
>  >> On 2020-05-09 21:01, John Mellor wrote:
>  >>> Now I need to wait for the next kernel upgrade, and see if that
>  >>> change fixes my crash/panic issue.
>  >> Of course you could boot to the previous kernel, remove the latest
>  >> one, reboot again to the previous
>  >> kernel and allow the upgrade to test.
>  >
>  > Many thanks, I think it worked.  The kernel upgrade this morning
>  > installed cleanly.
>  >
>  > This is probably a bug in the upgrade process, in that there is no check
>  > or code to remove the journal on /boot that was put there by an earlier
>  > fresh install.
>
>  It's not a bug.  There is no reason to remove the journal.  And for
>  anyone not using LVM, most will have /boot as part of / and removing the
>  journal from that would be a disaster.
>  _______________________________________________
>  users mailing list -- users(a)lists.fedoraproject.org
>  To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Samuel Sieb]

On 5/12/20 4:38 PM, John Mellor wrote:
>  Hi Samuel,
>  But that's the point.  A default install uses LVM, and therefore 
>  constructs a separate /boot partition running EXT4 with a journal.  You 
>  may of course decide to do your own partitions, but that setup is highly 
>  unusual and rare.  Most people will have the default partitioning, and 
>  therefore have the kernel upgrade bug. 
Except that I've never heard anyone complain about that.  It should not 
happen with ext4.  If it does, there's a problem.  Apparently btrfs is a 
little more lax about flushing the journal which has caused problems, 
but ext4 is not.  If you unmount or remount RO an ext4 filesystem, it 
will flush the journal before finishing.  So if you're finding that the 
journal has not been flushed, then you had an unclean unmount and need 
to find out why that's happening.

[Roger Heflin]

All it takes is one systemd bug, or any number of other conditions
that prevent a proper umount.

I have seen various kernel bugs cause it, I have seen the hw operating
the disks stop working.

In general using ext4+journal and/or xfs under the exact wrong
conditions can result in an unbootable system.

And those exact wrong conditions happen way too often, the hardware is
not perfect, VM datastores fill up, arrays and storage controllers go
offline randomly, nodes crash and/or power off and/or lose power
and/or overheat suddenly from both kernel bugs and bad hardware and/or
missed alarms.  If you design a system to fail too often in these
cases eventually the system will be abandoned for more robust setups.
  Saying worked as designed too many times for common conditions
results in a system that is not robust, and results in no one wanting
to use a given system.

On Tue, May 12, 2020 at 7:02 PM Samuel Sieb <samuel(a)sieb.net&gt; wrote:
>
> On 5/12/20 4:38 PM, John Mellor wrote:
> > Hi Samuel,
> > But that's the point.  A default install uses LVM, and therefore
> > constructs a separate /boot partition running EXT4 with a journal.  You
> > may of course decide to do your own partitions, but that setup is highly
> > unusual and rare.  Most people will have the default partitioning, and
> > therefore have the kernel upgrade bug.
>
> Except that I've never heard anyone complain about that.  It should not
> happen with ext4.  If it does, there's a problem.  Apparently btrfs is a
> little more lax about flushing the journal which has caused problems,
> but ext4 is not.  If you unmount or remount RO an ext4 filesystem, it
> will flush the journal before finishing.  So if you're finding that the
> journal has not been flushed, then you had an unclean unmount and need
> to find out why that's happening.
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Sam Varshavchik]

Roger Heflin writes:

>  What you are saying does not exactly match what I have previously
>  seen, but there is a known feature with using a journaling filesystem
>  (ext4-journal, or xfs) for /boot, if only the journal is updated and
>  if it is not yet replayed  into the non-journal then grub will not be
>  able to find the new files/updated files (grub filesystem code is
>  simple and does not process the journal so if critical updates are
>  still in the journal then those updates(changed file, new files)
>  cannot be seen).  To get this one generally has to do the update and
>  almost immediately reboot (within a few minutes though in some cases,
>  note syncing the does not replay the journal). 
I have used ext4 for /boot for 20+ years on Fedora, and can't say that I  
ever had this happen. And I always reboot after installing updates.

>  Long term the solution is to move boot to a non-journaled fs (ext
>  without a journal) or after each update umount/mount /boot(before
>  reboot).. 
I would be very much surprised if a regular reboot does not unmount /boot,  
prior to rebooting. I would think that all mounted filesystems except / will  
get explicitly unmounted just before a halt or a reboot.

[Ed Greshko]

On 2020-05-09 11:59, Sam Varshavchik wrote:
>  I would be very much surprised if a regular reboot does not unmount
> /boot, prior to rebooting. I would think that all mounted filesystems except / will
> get explicitly unmounted just before a halt or a reboot.  
You mean like this output from a reboot after a kernel update yesterday?

May 08 15:55:11 meimei.greshko.com systemd[1]: Unmounting /boot...
May 08 15:55:11 meimei.greshko.com systemd[1]: boot.mount: Succeeded.
May 08 15:55:11 meimei.greshko.com systemd[1]: Unmounted /boot.
May 08 15:55:11 meimei.greshko.com systemd[1]: systemd-reboot.service: Succeeded.
May 08 15:55:11 meimei.greshko.com systemd[1]: Finished Reboot.
May 08 15:55:11 meimei.greshko.com audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295
ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-reboot
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=?
terminal=? res=success'
May 08 15:55:11 meimei.greshko.com audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295
ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-reboot
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=?
terminal=? res=success'
May 08 15:55:11 meimei.greshko.com systemd[1]: Reached target Reboot.

-- 
The key to getting good answers is to ask good questions.

[Samuel Sieb]

On 5/8/20 8:59 PM, Sam Varshavchik wrote:
>  I have used ext4 for /boot for 20+ years on Fedora, and can't say
> that I 
>  ever had this happen. And I always reboot after installing updates. 
I've never had this happen over all the very many computers I've worked 
with.

> > Long term the solution is to move boot to a non-journaled fs
> (ext
> > without a journal) or after each update umount/mount /boot(before
> > reboot)..
>  
>  I would be very much surprised if a regular reboot does not unmount 
>  /boot, prior to rebooting. I would think that all mounted filesystems 
>  except / will get explicitly unmounted just before a halt or a reboot. 
And / gets remounted read-only, so it's effectively the same.

[Roger Heflin]

I never said that it does not get unmounted, but  unmounting does not
mean the journal got replayed.

Note the first thing ext4/xfs does on a mount/fsck is replays the
journal, the read-only mount would replay the journal, but indications
are that that fails often enough that the replay does not get done.
Also if you have /boot as a separate filesystem no ro,remount is done
so the journal is never replayed on /boot.  Overall the simplest is
just to use something without a journal.

https://www.gnu.org/software/grub/manual/grub/grub.html  see no 28:
grub has no code to write to the filesystem (and no journal replay).

https://bugzilla.redhat.com/show_bug.cgi?id=1569970  Note

On Fri, May 8, 2020 at 11:01 PM Sam Varshavchik <mrsam(a)courier-mta.com&gt; wrote:
>
> Roger Heflin writes:
>
> > What you are saying does not exactly match what I have previously
> > seen, but there is a known feature with using a journaling filesystem
> > (ext4-journal, or xfs) for /boot, if only the journal is updated and
> > if it is not yet replayed  into the non-journal then grub will not be
> > able to find the new files/updated files (grub filesystem code is
> > simple and does not process the journal so if critical updates are
> > still in the journal then those updates(changed file, new files)
> > cannot be seen).  To get this one generally has to do the update and
> > almost immediately reboot (within a few minutes though in some cases,
> > note syncing the does not replay the journal).
>
> I have used ext4 for /boot for 20+ years on Fedora, and can't say that I
> ever had this happen. And I always reboot after installing updates.
>
> > Long term the solution is to move boot to a non-journaled fs (ext
> > without a journal) or after each update umount/mount /boot(before
> > reboot)..
>
> I would be very much surprised if a regular reboot does not unmount /boot,
> prior to rebooting. I would think that all mounted filesystems except / will
> get explicitly unmounted just before a halt or a reboot.
>
>
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Sam Varshavchik]

Roger Heflin writes:

>  I never said that it does not get unmounted, but  unmounting does
> not
>  mean the journal got replayed.
>
>  Note the first thing ext4/xfs does on a mount/fsck is replays the
>  journal, the read-only mount would replay the journal, but indications
>  are that that fails often enough that the replay does not get done.
>  Also if you have /boot as a separate filesystem no ro,remount is done
>  so the journal is never replayed on /boot.  Overall the simplest is
>  just to use something without a journal. 
That's really strange.

Because, like I said, in 20+ years of installing kernels on multiple  
machines, all sorts of hardware, and immediately rebooting, this never  
happened to me.