Starting raid-check.timer renders system unusable

[Jonathan Dieter]

Hey everyone,

For reference, a bug report has been filed at
https://bugzilla.redhat.com/show_bug.cgi?id=1941335

I just wanted to give a heads up that I came across a bug today that
renders Fedora 33 systems unbootable, even after a clean install.  If
systemd starts raid-check.timer, it gets stuck in what looks like a
busy loop, is unable to start new services, systemctl stops responding
to commands, and we end up with a lot of zombie processes.

The problem is that raid-check.timer (a part of mdadm) is part of the
default boot process in Fedora Workstation, and the bug also exists on
the version of systemd installed with F33 GA.

I've tested on four different systems, each running F33 installed in
different ways (some upgrades, some fresh F33 installs), and on each of
them, starting raid-check.timer makes the system unusable.

There is a workaround: disabling raid-check.timer, but, if you can't
boot due to this bug, you have to boot into single-user mode (which
requires a root password to have been set).

Manually experimenting with dates seems to indicate that this bug is
triggered if the current date is after March 3rd, 2021 at 1:00AM, which
is why we haven't seen this bug before today.

I'm really hoping that I'm missing something obvious here, but I fear
that a good chunk of our Fedora systems will be unbootable if they're
rebooted without disabling raid-check.timer.

Jonathan

[Jonathan Dieter]

On Sun, 2021-03-21 at 17:52 +0000, Jonathan Dieter wrote:
>  There is a workaround: disabling raid-check.timer, but, if you
> can't
>  boot due to this bug, you have to boot into single-user mode (which
>  requires a root password to have been set). 
As Tom Hughes just pointed out to me, if you're stuck with an
unbootable system, you can add systemd.mask=raid-check.timer to the
kernel command line.

Jonathan

[Tom Hughes]

On 21/03/2021 17:52, Jonathan Dieter wrote:

>  There is a workaround: disabling raid-check.timer, but, if you
> can't
>  boot due to this bug, you have to boot into single-user mode (which
>  requires a root password to have been set). 
Adding systemd.mask=raid-check.timer to the kernel command line
when booting is another way to get past the hang.

Tom

-- 
Tom Hughes (tom(a)compton.nu)
http://compton.nu/

[Jonathan Dieter]

On Sun, 2021-03-21 at 17:52 +0000, Jonathan Dieter wrote:
>  I'm really hoping that I'm missing something obvious here,
> but I fear
>  that a good chunk of our Fedora systems will be unbootable if they're
>  rebooted without disabling raid-check.timer. 
Just to follow up on this, it appears that the problem is limited to
systems in the Europe/Dublin time zone.  Not good, but not the disaster
I was fearing.  Never been so glad to miss something obvious. ;)

Jonathan

[Ed Greshko]

On 22/03/2021 03:46, Jonathan Dieter wrote:
>  On Sun, 2021-03-21 at 17:52 +0000, Jonathan Dieter wrote:
> > I'm really hoping that I'm missing something obvious here, but I fear
> > that a good chunk of our Fedora systems will be unbootable if they're
> > rebooted without disabling raid-check.timer.
>  Just to follow up on this, it appears that the problem is limited to
>  systems in the Europe/Dublin time zone.  Not good, but not the disaster
>  I was fearing.  Never been so glad to miss something obvious. ;) 
Also, with raid-check.timer enabled I get....

[egreshko@f33g ~]$ sudo timedatectl set-timezone Europe/Dublin
[egreshko@f33g ~]$ date
Sun Mar 21 22:49:06 GMT 2021
[egreshko@f33g ~]$ sudo timedatectl set-timezone Asia/Taipei

Appears to hang at this point.  But a minute or so later....

Failed to set time zone: Connection timed out

-- 
People who believe they don't make mistakes have already made one.

[Chris Murphy]

On Sun, Mar 21, 2021 at 1:47 PM Jonathan Dieter <jdieter(a)gmail.com&gt; wrote:
>  On Sun, 2021-03-21 at 17:52 +0000, Jonathan Dieter wrote:
>  > I'm really hoping that I'm missing something obvious here, but I fear
>  > that a good chunk of our Fedora systems will be unbootable if they're
>  > rebooted without disabling raid-check.timer.
>
>  Just to follow up on this, it appears that the problem is limited to
>  systems in the Europe/Dublin time zone.  Not good, but not the disaster
>  I was fearing.  Never been so glad to miss something obvious. ;) 
Europe/Dublin is about to switch GMT->IST but likewise UK is about to
go GMT->BST so I'm wondering if the problem might actually pick up?

IRC user chrisawi points out that:
> today, raid-check.timer's trigger is Sun 2021-03-28 01:00:00
> yesterday, it was Sun 2021-03-21 01:00:00 
And raid-check.timer contains:
OnCalendar=Sun *-*-* 01:00:00

The switch from GMT->IST and GMT->BST happen on 2021-03-28 01:00:00

Conversely in most of Europe, it will change 2021-03-28 but with two
differences that might be relevant: 02:00:00, and from Standard to
Daylight time.



-- 
Chris Murphy