It's nice if the clock is more-or-less right before services start.
bind/named resolver doesn't resolve if the clock is horribly out (DNSSEC?) and it doesn't fix itself after a systemctl restart chronyd, you need to actually bounce named. Without named things like 'ntp.pool.fedora' don't work, so there's a catch-22.
postgresql can be unhappy on boot. This is clock related, but I can't replicate it 100% yet.
journald will rotate logs as soon as chronyd syncs the clock. So any logs from bootup (1970) are lost. That's another reason to get the clock +- correct as soon as possible.
I'm currently using a cron script to touch a file every 10 minutes, and read that on bootup (before chronyd), and I've added a 'Requires=touchClock' to some systemd services.
The script is smart enough to refuse to run if it'll move the time backwards.
On Thu, Sep 24, 2015 at 06:18:24PM +0200, Berend De Schouwer wrote:
It's nice if the clock is more-or-less right before services start.
There was a discussion about this here few weeks ago. I think there are several options how could it be improved in Fedora.
A) add a "fake-hwclock" service to restore time across reboot 1) disabled by default 2) enabled by default on arm 3) enabled only if there is no RTC or it's missing battery 4) ask the user in the installer B) restore time from the driftfile in chronyd (if it's in future) and start it earlier in the boot 1) configure it to do that by default on arm 2) modify it to always try to detect if the time is "uninitialized"
As for the detection, would it be sufficient to check if the date is 1970-01-01 or 2000-01-01? Are there any RTCs with which the kernel would boot to a different date?
Thoughts?
On Fri, 2015-09-25 at 10:53 +0200, Miroslav Lichvar wrote:
On Thu, Sep 24, 2015 at 06:18:24PM +0200, Berend De Schouwer wrote:
It's nice if the clock is more-or-less right before services start.
There was a discussion about this here few weeks ago. I think there are several options how could it be improved in Fedora.
A) add a "fake-hwclock" service to restore time across reboot 1) disabled by default 2) enabled by default on arm 3) enabled only if there is no RTC or it's missing battery
Is there a syscall for RTC? Ideally that's what's needed. My system's kernel has a nortc command line option, so the kernel has been told, and it's in /proc/cmdline.
4) ask the user in the installer B) restore time from the driftfile in chronyd (if it's in future) and start it earlier in the boot
There absolutely must be a solution earlier in the boot. Probably right after mount /. It needs to be in the right ballpark before any services that use certificates start, and those are becoming more prevalent.
I tried playing with time-sync.target; but that didn't work. Maybe add it to sysinit.target.wants?
1) configure it to do that by default on arm 2) modify it to always try to detect if the time is "uninitialized"
In ntp (dunno about chronyd) the driftfile is there to find small changes, and it's purpose is to help if network connectivity is lost.
For example, if network connectivity is lost for 24 hours, you'd like to know if it's 23:59:55 or 24:00:05.
Adding a 45 year drift every now and then would likely break the driftfile. It'll record 0.001 ms every second, and every now and then 45 years.
As for the detection, would it be sufficient to check if the date is 1970-01-01 or 2000-01-01? Are there any RTCs with which the kernel would boot to a different date?
I've had other dates on broken motherboards. I'm pretty sure I've seen 1980 on a motherboard with a dead battery, but it was manufactured before 2000. I don't think catering for broken hardware is required, but it might be naive to assume 1970 or 2000.
There's a timestamp on all the filesystems. There's at least "last successful mount" for fsck. The mount time is likely to be 1970, since the clock resets, but maybe mtime or atime on / is a good check. If the RTC is more than a day before mtime on /, the RTC is wrong. It needs to be at least 24 hours to take care of broken BIOS timezone, possibly 25 hours to cater for daylight savings. It also needs 24 hours for people who fly across the dateline.
For this specific case, I used one year. If the filesystem is more than a year ahead of the RTC, assume the RTC is wrong.
For this specific case I used /var/lib/clock/clock since I wasn't sure if atime on / was strictly incrementing.
On Thu, Sep 24, 2015 at 06:18:24PM +0200, Berend De Schouwer wrote:
I'm currently using a cron script to touch a file every 10 minutes, and read that on bootup (before chronyd), and I've added a 'Requires=touchClock' to some systemd services.
I think you don't need the cron script. If you make the (reasonable?) assumption that files in /var/log are updated regularly then:
# find /var/log -type f -print0 | xargs -0 stat -c '%Y %n' /var/log | sort -nr | head -1
(You can omit the %n if you don't care about the actual file that is the newest).
Generally, I think your plan is a good one! Would be good for my CubieTruck too. How about trying to get it into systemd upstream?
Rich.
On Sun, Sep 27, 2015 at 09:53:14PM +0100, Richard W.M. Jones wrote:
# find /var/log -type f -print0 | xargs -0 stat -c '%Y %n' /var/log | sort -nr | head -1
A stray "/var/log" crept in there (but doesn't change the output). The correct command should be:
# find /var/log -type f -print0 | xargs -0 stat -c '%Y %n' | sort -nr | head -1
Rich.
On Sun, 2015-09-27 at 21:53 +0100, Richard W.M. Jones wrote:
On Thu, Sep 24, 2015 at 06:18:24PM +0200, Berend De Schouwer wrote:
I'm currently using a cron script to touch a file every 10 minutes, and read that on bootup (before chronyd), and I've added a 'Requires=touchClock' to some systemd services.
I think you don't need the cron script. If you make the (reasonable?) assumption that files in /var/log are updated regularly then:
# find /var/log -type f -print0 | xargs -0 stat -c '%Y %n' /var/log | sort -nr | head -1
(You can omit the %n if you don't care about the actual file that is the newest).
I think systemd systems are supposed to move to journald. Are there still official /var/log files that are intended to remain over the next few Fedora releases, and guaranteed to exist and updated?
I'm not convinced that depending on a wildcard file is robust long- term. It sounds like a recipe for random failures.
I'm don't think depending on journald is right. Journald starts on initrd before the filesystem is mounted, and then moves the journal after / is mounted; so it might be (not confirmed) that /var/log/journal/ has a bad timestamp on boot.
Generally, I think your plan is a good one!
I can't claim credit. I got the idea from tedu (BSD). There are no original ideas.
Would be good for my CubieTruck too. How about trying to get it into systemd upstream?
I think we should. Especially since it already ships systemd- timesyncd. How do we go about that?
Do I go on the systemd mailing list? OK, away we go...
On Tue, Sep 29, 2015 at 06:55:00AM +0200, Berend De Schouwer wrote:
On Sun, 2015-09-27 at 21:53 +0100, Richard W.M. Jones wrote:
On Thu, Sep 24, 2015 at 06:18:24PM +0200, Berend De Schouwer wrote:
I'm currently using a cron script to touch a file every 10 minutes, and read that on bootup (before chronyd), and I've added a 'Requires=touchClock' to some systemd services.
I think you don't need the cron script. If you make the (reasonable?) assumption that files in /var/log are updated regularly then:
# find /var/log -type f -print0 | xargs -0 stat -c '%Y %n' /var/log | sort -nr | head -1
(You can omit the %n if you don't care about the actual file that is the newest).
I think systemd systems are supposed to move to journald. Are there still official /var/log files that are intended to remain over the next few Fedora releases, and guaranteed to exist and updated?
journald stores files under /var/log ...
I'm not convinced that depending on a wildcard file is robust long- term. It sounds like a recipe for random failures.
I'm don't think depending on journald is right. Journald starts on initrd before the filesystem is mounted, and then moves the journal after / is mounted; so it might be (not confirmed) that /var/log/journal/ has a bad timestamp on boot.
... and this doesn't matter because the command above finds the newest file under /var/log. If journald writes a file with a 1970-01-01 timestamp it won't be the newest file.
I think we should. Especially since it already ships systemd- timesyncd. How do we go about that?
But anyway as systemd-timesyncd was pointed out in the other reply, and it seems to be doing mostly the same as your technique, I guess we need to find out if we can use that (probably it's just a matter of enabling the service?). systemd has a mailing list:
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Rich.
On Tue, Sep 29, 2015 at 08:34:44AM +0100, Richard W.M. Jones wrote:
But anyway as systemd-timesyncd was pointed out in the other reply, and it seems to be doing mostly the same as your technique, I guess we need to find out if we can use that (probably it's just a matter of enabling the service?). systemd has a mailing list:
I read your other reply now about systemd-timesyncd running too late.
Perhaps the answer is to split the two functions of systemd-timesyncd into two services? The one for updating the time based on a newest file timestamp can run before / independent of the network.
Something to discuss on systemd-devel anyway.
Rich.
On Tue, Sep 29, 2015 at 08:36:55AM +0100, Richard W.M. Jones wrote:
On Tue, Sep 29, 2015 at 08:34:44AM +0100, Richard W.M. Jones wrote:
But anyway as systemd-timesyncd was pointed out in the other reply, and it seems to be doing mostly the same as your technique, I guess we need to find out if we can use that (probably it's just a matter of enabling the service?).
chronyd can do that too, with the -s option it will set time to the modification time of the drift file if there is no RTC and it can be configured to ignore the RTC if it doesn't have battery, but this is not enabled by default.
Perhaps the answer is to split the two functions of systemd-timesyncd into two services? The one for updating the time based on a newest file timestamp can run before / independent of the network.
That would be a good idea. Another possibility is to set the system time in initramfs to the last mount time of the root filesystem. On some systems this is enabled with the "fixrtc" kernel option.
On Tue, 2015-09-29 at 08:36 +0100, Richard W.M. Jones wrote:
On Tue, Sep 29, 2015 at 08:34:44AM +0100, Richard W.M. Jones wrote:
But anyway as systemd-timesyncd was pointed out in the other reply, and it seems to be doing mostly the same as your technique, I guess we need to find out if we can use that (probably it's just a matter of enabling the service?). systemd has a mailing list:
I read your other reply now about systemd-timesyncd running too late.
I've tried F23 Beta on my Raspberry Pi, and systemd-timesyncd works fine now. It detects the clock rewind, and resets it.
I did try to get a Changelog between F21 and F23 for systemd-timesyncd to try and find what changed, but I can't.
I was going to upgrade anyway, but the reason I did it during Beta, was that the sd-card in the Pi popped. So I can't go back to F21 and confirm or diagnose the problem.
It might be a good idea in Arm remixes to prefer systemd-timesyncd to chronyd for this reason. Or try chronyd with -s.
On Thu, Sep 24, 2015 at 11:18 AM, Berend De Schouwer < berend.de.schouwer@gmail.com> wrote:
It's nice if the clock is more-or-less right before services start.
Isn't this what systemd-timesyncd is for?
http://www.freedesktop.org/software/systemd/man/systemd-timesyncd.service.ht...
On Mon, 2015-09-28 at 10:48 -0500, Jeffrey Ollie wrote:
On Thu, Sep 24, 2015 at 11:18 AM, Berend De Schouwer <berend.de.schou wer@gmail.com> wrote:
It's nice if the clock is more-or-less right before services start.
Isn't this what systemd-timesyncd is for?
http://www.freedesktop.org/software/systemd/man/systemd-timesyncd.ser vice.html
That actually acts too late, and depends on a network. It fails to help if the network is down at boot, unless you make all the services block until the network is up.
It also fails if named doesn't work, and named doesn't work if the clock is horribly out. So now you have a catch-22. Named depends on a clock, and the clock depends on working DNS.
I think timesyncd is supposed to replace chronyd.
-
Berend De Schouwer -
Jeffrey Ollie -
Miroslav Lichvar -
Richard W.M. Jones