Some folks are reporting a hang when booting Fedora 33 this morning (or also possibly wake from sleep). We're still digging into it but looks like a work around right now is to boot with parameter: systemd.mask=raid-check.timer
You can subsequently 'systemctl disable raid-check.timer'. Folks with md raid setups might want to set a reminder to enable the timer again once the issue is sorted out.
On 22/03/2021 02:27, Chris Murphy wrote:
Some folks are reporting a hang when booting Fedora 33 this morning (or also possibly wake from sleep). We're still digging into it but looks like a work around right now is to boot with parameter: systemd.mask=raid-check.timer
You can subsequently 'systemctl disable raid-check.timer'. Folks with md raid setups might want to set a reminder to enable the timer again once the issue is sorted out.
It has been noted on the devel list that it only seems to fail if the time zone is set to Europe/Dublin.
Also, FWIW, when I used timedatectl to set my time zone to Europe/Dublin my system hung when doing an "init 6".
On Mon, 2021-03-22 at 06:12 +0800, Ed Greshko wrote:
On 22/03/2021 02:27, Chris Murphy wrote:
Some folks are reporting a hang when booting Fedora 33 this morning (or also possibly wake from sleep). We're still digging into it but looks like a work around right now is to boot with parameter: systemd.mask=raid-check.timer
You can subsequently 'systemctl disable raid-check.timer'. Folks with md raid setups might want to set a reminder to enable the timer again once the issue is sorted out.
It has been noted on the devel list that it only seems to fail if the time zone is set to Europe/Dublin.
Also, FWIW, when I used timedatectl to set my time zone to Europe/Dublin my system hung when doing an "init 6".
Interesting. Does this happen with Europe/London (which is the exact same timezone)?
poc
On 22/03/2021 06:36, Patrick O'Callaghan wrote:
On Mon, 2021-03-22 at 06:12 +0800, Ed Greshko wrote:
On 22/03/2021 02:27, Chris Murphy wrote:
Some folks are reporting a hang when booting Fedora 33 this morning (or also possibly wake from sleep). We're still digging into it but looks like a work around right now is to boot with parameter: systemd.mask=raid-check.timer
You can subsequently 'systemctl disable raid-check.timer'. Folks with md raid setups might want to set a reminder to enable the timer again once the issue is sorted out.
It has been noted on the devel list that it only seems to fail if the time zone is set to Europe/Dublin.
Also, FWIW, when I used timedatectl to set my time zone to Europe/Dublin my system hung when doing an "init 6".
Interesting. Does this happen with Europe/London (which is the exact same timezone)?
I just tested and the answer is "no".
Another interesting factoid.
[egreshko@f33g ~]$ sudo timedatectl set-timezone Europe/Dublin [egreshko@f33g ~]$ date Sun Mar 21 22:49:06 GMT 2021 [egreshko@f33g ~]$ sudo timedatectl set-timezone Asia/Taipei
And the command is "hung"
Until....
Failed to set time zone: Connection timed out
On Mon, 2021-03-22 at 06:54 +0800, Ed Greshko wrote:
On 22/03/2021 06:36, Patrick O'Callaghan wrote:
On Mon, 2021-03-22 at 06:12 +0800, Ed Greshko wrote:
On 22/03/2021 02:27, Chris Murphy wrote:
Some folks are reporting a hang when booting Fedora 33 this morning (or also possibly wake from sleep). We're still digging into it but looks like a work around right now is to boot with parameter: systemd.mask=raid-check.timer
You can subsequently 'systemctl disable raid-check.timer'. Folks with md raid setups might want to set a reminder to enable the timer again once the issue is sorted out.
It has been noted on the devel list that it only seems to fail if the time zone is set to Europe/Dublin.
Also, FWIW, when I used timedatectl to set my time zone to Europe/Dublin my system hung when doing an "init 6".
Interesting. Does this happen with Europe/London (which is the exact same timezone)?
I just tested and the answer is "no".
Another interesting factoid.
[egreshko@f33g ~]$ sudo timedatectl set-timezone Europe/Dublin [egreshko@f33g ~]$ date Sun Mar 21 22:49:06 GMT 2021 [egreshko@f33g ~]$ sudo timedatectl set-timezone Asia/Taipei
And the command is "hung"
Until....
Failed to set time zone: Connection timed out
Clearly BZ-worthy I'd say.
poc
On 22/03/2021 07:07, Patrick O'Callaghan wrote:
Clearly BZ-worthy I'd say.
Oh, there is one. https://bugzilla.redhat.com/show_bug.cgi?id=1941335
Ed Greshko writes:
Another interesting factoid.
[egreshko@f33g ~]$ sudo timedatectl set-timezone Europe/Dublin [egreshko@f33g ~]$ date Sun Mar 21 22:49:06 GMT 2021 [egreshko@f33g ~]$ sudo timedatectl set-timezone Asia/Taipei
And the command is "hung"
Until....
Failed to set time zone: Connection timed out
I recall the days when to set the system timezone one only needed to update /etc/timezone
echo EST5EDT >/etc/timezone
or
echo America/New_York >/etc/timezone
Presto, and your default timezone has changed.
But, it seems that this was too simple. Things cannot be this simple anymore. We must have daemons involved in this process, for some unclear reason. The more, the merrier.
On Sun, Mar 21, 2021 at 4:37 PM Patrick O'Callaghan pocallaghan@gmail.com wrote:
On Mon, 2021-03-22 at 06:12 +0800, Ed Greshko wrote:
On 22/03/2021 02:27, Chris Murphy wrote:
Some folks are reporting a hang when booting Fedora 33 this morning (or also possibly wake from sleep). We're still digging into it but looks like a work around right now is to boot with parameter: systemd.mask=raid-check.timer
You can subsequently 'systemctl disable raid-check.timer'. Folks with md raid setups might want to set a reminder to enable the timer again once the issue is sorted out.
It has been noted on the devel list that it only seems to fail if the time zone is set to Europe/Dublin.
Also, FWIW, when I used timedatectl to set my time zone to Europe/Dublin my system hung when doing an "init 6".
Interesting. Does this happen with Europe/London (which is the exact same timezone)?
Possibly. And perhaps not because of the timezone but when the change is scheduled to happen: 2021-03-28 01:00:00
That's the same datetime as this particular timer: OnCalendar=Sun *-*-* 01:00:00
That translates as Sun 2021-03-28 01:00:00 as of today. Yesterday that was Sun 2021-03-21 01:00:00 which is probably why the issue didn't show up last week.
Further, it might be true that other Europe timezones aren't affected because their official time change happens at 02:00:00.
On 22/03/2021 19:26, Patrick O'Callaghan wrote:
On Sun, 2021-03-21 at 19:48 -0400, Sam Varshavchik wrote:
But, it seems that this was too simple. Things cannot be this simple anymore. We must have daemons involved in this process, for some unclear reason. The more, the merrier.
Pandemonium.
Even though these comments are not germane to the subject it should be pointed out that the command "timedatectl" does more than just set the timezone. It does unify, or seeks to unify, aspects of setting the time and date.
An example of that is that the user need not know what process is being used for NTP service. The set-ntp takes care of that.
When it comes to time zones, it can help clear up some ambiguities for people that don't live in the US. Like, did you know that Taiwan the abbreviation for our time zone is CST? Setting the time zone from the list provided by --list-timezones can assist those unaware of such things.
On Mon, 2021-03-22 at 19:43 +0800, Ed Greshko wrote:
When it comes to time zones, it can help clear up some ambiguities for people that don't live in the US. Like, did you know that Taiwan the abbreviation for our time zone is CST? Setting the time zone from the list provided by --list-timezones can assist those unaware of such things.
So does tzselect, which long predates systemd.
poc
On 22/03/2021 20:07, Patrick O'Callaghan wrote:
On Mon, 2021-03-22 at 19:43 +0800, Ed Greshko wrote:
When it comes to time zones, it can help clear up some ambiguities for people that don't live in the US. Like, did you know that Taiwan the abbreviation for our time zone is CST? Setting the time zone from the list provided by --list-timezones can assist those unaware of such things.
So does tzselect, which long predates systemd.
tzselect does one thing. The thrust of my comment is that timedatectl does that (as opposed to the "echo") as well as other things.
But, again, since not germane it is best to just drop this portion of the thread.
Ed Greshko writes:
On 22/03/2021 19:26, Patrick O'Callaghan wrote:
On Sun, 2021-03-21 at 19:48 -0400, Sam Varshavchik wrote:
But, it seems that this was too simple. Things cannot be this simple anymore. We must have daemons involved in this process, for some unclear reason. The more, the merrier.
Pandemonium.
Even though these comments are not germane to the subject it should be pointed out that the command "timedatectl" does more than just set the timezone. It does unify, or seeks to unify, aspects of setting the time and date.
What's there beyond a single setting, somewhere, that designates the system's time zone, that's not clear to me. It all boils down to a single setting, somewhere.
And just having a single file with the setting is the most simple, basic, approach that could possibly be. Well, I do see an argument for a configuration file, somewhere, with a larger scope and the timezone is one configuration setting in there. That also seems reasonable to me.
But now there's an entire executable. That apparently talks to some daemon somewhere. And, the end result is a complicated process that shouldn't be more involved than just saving the setting somewhere.
It's going to be pretty hard to have
echo Asia/Taipei >/etc/timezone
result in a system hanging.
An example of that is that the user need not know what process is being used for NTP service. The set-ntp takes care of that.
I'm not sure why NTP has to be involved in any of this. NTP counts seconds. The timezone setting determines how this single counter translates into wall time. I don't see how changing the system's timezone would have any effect on ntp.
When it comes to time zones, it can help clear up some ambiguities for people that don't live in the US. Like, did you know that Taiwan the abbreviation for our time zone is CST? Setting the time zone from the list provided by --list-timezones can assist those unaware of such things.
I suppose it never occured to anyone to have that list kept in a plain text file somewhere.
I can see the merit of using a command to the timezone if all it did was check the given timezone against the list stored in such a configuration file, and then update /etc/timezone if the timezone setting is valid.
I take it that it's still unknown what's causing this hang. I don't think it matters. The real bug is having this rube-goldbergian contraption in the first place. Just for setting a time zone.
This is complexity purely for the sake of having complexity, and introducing more unneeded dependencies on the systemd monstrosity.
On Mon, 22 Mar 2021 08:22:46 -0400 Sam Varshavchik wrote:
This is complexity purely for the sake of having complexity, and introducing more unneeded dependencies on the systemd monstrosity.
Didn't this thread start with a warning about raid systems hanging at boot? What on earth does raid care what the timezone is? Does some raid header somewhere have timestamps recorded in local time to emulate Microsoft? :-).
On Mon, Mar 22, 2021 at 9:52 AM Tom Horsley horsley1953@gmail.com wrote:
Didn't this thread start with a warning about raid systems hanging at boot? What on earth does raid care what the timezone is?
From the thread on the devel list[1], it seems that the issue may be that Ireland switched to summer time at 0100, which is also when the timer is set to run. Most jurisdictions change at 0200, so that may be why it's not more widespread (since most of the Americas changed a week ago).
[1] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
On Mon, 22 Mar 2021 at 10:45, Tom Horsley horsley1953@gmail.com wrote:
On Mon, 22 Mar 2021 08:22:46 -0400 Sam Varshavchik wrote:
This is complexity purely for the sake of having complexity, and
introducing
more unneeded dependencies on the systemd monstrosity.
Didn't this thread start with a warning about raid systems hanging at boot? What on earth does raid care what the timezone is? Does some raid header somewhere have timestamps recorded in local time to emulate Microsoft? :-).
There is a long history of problems going back to UNIX caused by clock adjustments that result in skipping over the time for some process to run via cron or at, etc. There are countermeasures to handle this, but I suspect it is one of those simple problems that does not have any robust solutions. I used schedule stuff for "random" minutes, never right on the hour.
On Mon, 2021-03-22 at 10:06 -0400, Ben Cotton wrote:
On Mon, Mar 22, 2021 at 9:52 AM Tom Horsley horsley1953@gmail.com wrote:
Didn't this thread start with a warning about raid systems hanging at boot? What on earth does raid care what the timezone is?
From the thread on the devel list[1], it seems that the issue may be that Ireland switched to summer time at 0100, which is also when the timer is set to run. Most jurisdictions change at 0200, so that may be why it's not more widespread (since most of the Americas changed a week ago).
Together with the rest of Europe, Ireland has not yet switched and won't until next weekend, so I don't see why it should be triggering now.
Also, the switch time of 0100 is the same as the UK.
poc
The bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1941335
Fix is nearly ready: https://github.com/systemd/systemd/pull/19075
-- Chris Murphy
Tom Horsley writes:
On Mon, 22 Mar 2021 08:22:46 -0400 Sam Varshavchik wrote:
This is complexity purely for the sake of having complexity, and
introducing
more unneeded dependencies on the systemd monstrosity.
Didn't this thread start with a warning about raid systems hanging at boot? What on earth does raid care what the timezone is? Does some raid header somewhere have timestamps recorded in local time to emulate Microsoft? :-).
You're preaching to the choir.
If setting the timezone was no more than
echo 'Asia/Seoul' >/etc/timezone
then how could this, in any way, end up breaking RAID? Even if that was done at the very exact time the raid daemon woke up?
I just saw, in the other message, the link to the github pull request to fix this. I rolled the dice and decided to look at it, and spend a few moments trying to figure what's up. It seems to be trying to derive epoch time for the next start time, something along those lines.
The existing algorithm fails and results in the infinite loop. The fix appears to be not to actually fix the algorithm, but to cap the number of iteration to 1000. And then there were also four more commits thrown in for a good measure, making various changes that seems to be somewhat random and unrelated to the bug. Hopefully, the first commit is good enough to fix the bug, and the rest won't create four new bugs in its place.
What struck me is this: the format of the zone files is documented. It's tzfile(5). I wrote a parser for it, some time ago. By parsing the zone file you would know when the timezone offset changes *and in which direction*. You can factor this in, when doing epoch time calculations. It would be nice if glibc exposed this data. It doesn't, but it's not hard to get at it yourself. It's puzzling why systemd didn't reinvent this wheel, given all the wheels it already reinvented.
Really, this should've been a deterministic calculation: compute the next run time, based on the calendar spec and the current time, and ignoring time shifts. Now, you've computed the epoch time for the next run time, on the assumption that there are no timezone changes. And what about them, well, you have a nice list of them, in the tzfile(5).
So, then you go through the tzfile(5), which gives you all the timezone changes between the two epoch times, and use the deltas to adjust your epoch time. And you end up at the right time. And if you land in the black hole, you'll just end up running the task at some particular point in time, and then regroup on the next run. And none of this heuristic guesswork to derive the epoch time, that has sooooo many opportunities to go sideways. Which it did.
On Mon, 22 Mar 2021 18:07:12 -0400 Sam Varshavchik wrote:
It seems to be trying to derive epoch time for the next start time, something along those lines.
More obviously it seems to be systemd re-implementing cron from scratch instead of just using the already working after decades of use cron itself.
Tom Horsley writes:
On Mon, 22 Mar 2021 18:07:12 -0400 Sam Varshavchik wrote:
It seems to be trying to derive epoch time for the next start time, something along those lines.
More obviously it seems to be systemd re-implementing cron from scratch instead of just using the already working after decades of use cron itself.
Yup. This is an obvious vendor lock-in play. Replace widely-known technologies with new, arcane stuff whose full documentation is kept internal, and is leveraged as a value-added product.
On 3/22/21 6:00 PM, Sam Varshavchik wrote:
Tom Horsley writes:
On Mon, 22 Mar 2021 18:07:12 -0400 Sam Varshavchik wrote:
It seems to be trying to derive epoch time for the next start time, something along those lines.
More obviously it seems to be systemd re-implementing cron from scratch instead of just using the already working after decades of use cron itself.
Yup. This is an obvious vendor lock-in play. Replace widely-known technologies with new, arcane stuff whose full documentation is kept internal, and is leveraged as a value-added product.
We all understand by now that you really don't like systemd, but please keep your conspiracy theories to yourself. They don't even make sense. Who's the vendor? It's fully open-source software, where's the lock-in? Sometimes things do need to be re-implemented, there's nothing wrong with that. The documentation is reasonably good and I'm pretty sure there's no secret documentation stash somewhere. So enough with the systemd hate on the mailing list. Take it to facebook or reddit or even better, keep it to yourself.
Samuel Sieb writes:
On 3/22/21 6:00 PM, Sam Varshavchik wrote:
Yup. This is an obvious vendor lock-in play. Replace widely-known technologies with new, arcane stuff whose full documentation is kept internal, and is leveraged as a value-added product.
We all understand by now that you really don't like systemd, but please keep your conspiracy theories to yourself. They don't even make sense. Who's the vendor?
Ummm… IBM?
It's fully open-source software, where's the lock-in?
Here's a hundred thousand lines of code. Documentation? Not so much. But, hey, it's "open-source".
Sometimes things do need to be re-implemented, there's nothing wrong with that.
You wrote the magic word: "need". You hit the nail right on its head! So, cron needed to be reimplemented …exactly why?
Heck, cron was almost a portable standard of its own. We went from vixie- cron to cronie, with barely a hiccup. crontabs worked the same way, and they've been working the same way not just for years, but for decades.
-
Ben Cotton -
Chris Murphy -
Ed Greshko -
George N. White III -
Patrick O'Callaghan -
Sam Varshavchik -
Samuel Sieb -
Tom Horsley