I have chrony running and pointing to my ISP's time server, not the Fedora pool, so it's always the same NTP server.
This box is up 24/7, and I just rebooted it. And I get this, after a reboot:
Nov 23 08:53:10 shorty chronyd[992]: System clock TAI offset set to 37 seconds Nov 23 08:53:10 shorty chronyd[992]: System clock wrong by -5.652285 seconds, adjustment started Nov 23 08:53:10 shorty chronyd[992]: System clock was stepped by -5.652285 seconds
This was a reboot, and not a shutdown.
I believe that something should be saving the system time to the hardware clock, so after a reboot things are more or less where they are, and this looks like is not happening. I should not have chrony adjust time by that much after a reboot. Anyone know where to investigate this further?
On Sat, 23 Nov 2019 08:59:02 -0500 Sam Varshavchik wrote:
Anyone know where to investigate this further?
The linux app that syncs to the hardware is "hwclock" (which has various obscure parameters described in the man page).
You could make a script to sync with hwclock then reboot and see if the time is more in sync when it comes back. That would verify it isn't being synced (but, unfortunately, wouldn't help track down where the hwclock call ought to happen).
Or maybe just replace the cmos battery on the motherboard and see if it get better :-).
On 2019-11-23 22:22, Tom Horsley wrote:
On Sat, 23 Nov 2019 08:59:02 -0500 Sam Varshavchik wrote:
Anyone know where to investigate this further?
The linux app that syncs to the hardware is "hwclock" (which has various obscure parameters described in the man page).
You could make a script to sync with hwclock then reboot and see if the time is more in sync when it comes back. That would verify it isn't being synced (but, unfortunately, wouldn't help track down where the hwclock call ought to happen).
Maybe add an ExecStop= option to the chronyd.service with the appropriate parameters?
FWIW, on my rather old HP system I too get "System clock wrong" on some (not all) reboots. A quick look and I estimate the average for this system is about 6 seconds.
The discrepancy is noted within 2 seconds of boot. I wouldn't have noticed it if Sam hadn't pointed it out.
CMOs battery on the motherboard?
On 23 Nov 2019, at 14:00, Sam Varshavchik mrsam@courier-mta.com wrote:
I have chrony running and pointing to my ISP's time server, not the Fedora pool, so it's always the same NTP server.
This box is up 24/7, and I just rebooted it. And I get this, after a reboot:
Nov 23 08:53:10 shorty chronyd[992]: System clock TAI offset set to 37 seconds Nov 23 08:53:10 shorty chronyd[992]: System clock wrong by -5.652285 seconds, adjustment started Nov 23 08:53:10 shorty chronyd[992]: System clock was stepped by -5.652285 seconds
This was a reboot, and not a shutdown.
I believe that something should be saving the system time to the hardware clock, so after a reboot things are more or less where they are, and this looks like is not happening. I should not have chrony adjust time by that much after a reboot. Anyone know where to investigate this further?
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Tom Horsley writes:
On Sat, 23 Nov 2019 08:59:02 -0500 Sam Varshavchik wrote:
Anyone know where to investigate this further?
The linux app that syncs to the hardware is "hwclock" (which has various obscure parameters described in the man page).
You could make a script to sync with hwclock then reboot and see if the time is more in sync when it comes back. That would verify it isn't being synced (but, unfortunately, wouldn't help track down where the hwclock call ought to happen).
I would expect that something would already be doing that.
Or maybe just replace the cmos battery on the motherboard and see if it get better :-).
Checking some other boxes I just rebooted, only this one is griping about the clock.
All these boxes are up 24/7. The longest they've been without power is about a week, when power's been out after a hurricane. Plus an occasional day of downtime, every other year or so, when power goes out for other reasons.
Some of those boxes are just as old as this one. Something has to be syncing the system time to the hardware clock on a reboot, otherwise their clocks would've drifted just as much. Maybe the other motherboards don't drain the battery at all when system power is on, and this one does.
So, hwclock must be getting synced. But I don't see where hwclock would be getting called from. grepping /lib/systemd/system finds nothing. hwclock itself comes from util-linux, which doesn't install anything there.
On Sat, 2019-11-23 at 10:09 -0500, Sam Varshavchik wrote:
Tom Horsley writes:
On Sat, 23 Nov 2019 08:59:02 -0500 Sam Varshavchik wrote:
Anyone know where to investigate this further?
The linux app that syncs to the hardware is "hwclock" (which has various obscure parameters described in the man page).
You could make a script to sync with hwclock then reboot and see if the time is more in sync when it comes back. That would verify it isn't being synced (but, unfortunately, wouldn't help track down where the hwclock call ought to happen).
I would expect that something would already be doing that.
Or maybe just replace the cmos battery on the motherboard and see if it get better :-).
Checking some other boxes I just rebooted, only this one is griping about the clock.
All these boxes are up 24/7. The longest they've been without power is about a week, when power's been out after a hurricane. Plus an occasional day of downtime, every other year or so, when power goes out for other reasons.
Some of those boxes are just as old as this one. Something has to be syncing the system time to the hardware clock on a reboot, otherwise their clocks would've drifted just as much. Maybe the other motherboards don't drain the battery at all when system power is on, and this one does.
So, hwclock must be getting synced. But I don't see where hwclock would be getting called from. grepping /lib/systemd/system finds nothing. hwclock itself comes from util-linux, which doesn't install anything there.
Is chrony itself supposed to keep the hw clock synced? From https://chrony.tuxfamily.org/manual.html#rtcsync-directive::
4.2.55 rtcsync
The rtcsync directive enables a mode where the system time is periodically copied to the real time clock (RTC).
On Linux the RTC copy is performed by the kernel every 11 minutes. This directive cannot be used when the normal RTC tracking is enabled, i.e. when the rtcfile directive is used.
On Mac OS X, chronyd will perform the RTC copy every 60 minutes when the system clock is in a synchronised state.
On other systems this directive does nothing.
/Louis
On 2019-11-23 21:59, Sam Varshavchik wrote:
Anyone know where to investigate this further?
Suggest you keep an eye on the drift by doing something like
date +%Y-%m-%d\ %H:%M:%S.%N > tdiff ; hwclock -r >> tdiff
on a periodic basis.
I would think it would be a good idea to do it just before a reboot and then compare the difference with that reported in the journal.
Louis Lagendijk writes:
On Sat, 2019-11-23 at 10:09 -0500, Sam Varshavchik wrote:
So, hwclock must be getting synced. But I don't see where hwclock would be getting called from. grepping /lib/systemd/system finds nothing. hwclock itself comes from util-linux, which doesn't install anything there.
Is chrony itself supposed to keep the hw clock synced? From https://chrony.tuxfamily.org/manual.html#rtcsync-directive::
4.2.55 rtcsync
The rtcsync directive enables a mode where the system time is periodically copied to the real time clock (RTC).
That looks like it. The rtcsync directive exist in my chrony.conf. I didn't put there, so it must be in the default chrony.conf
So, the hardware clock seems to be getting synced, but it loses some seconds during a reboot.
This server gets rebooted on a regular schedule, and the default rotation of /var/log/messages captured last five reboots, and showed that only the three of them resulted in a skewed system clock:
Oct 20 09:43:00 shorty chronyd[1080]: System clock wrong by -13.429737 seconds, adjustment started Nov 16 06:50:35 shorty chronyd[1068]: System clock wrong by -6.621246 seconds, adjustment started Nov 23 08:53:10 shorty chronyd[992]: System clock wrong by -5.652285 seconds, adjustment started
The logs of chronyd starting on Nov 2nd and Nov 9nd did not log any time adjustment.
Next time I cycle it, I'll check what the hwclock reports, before the reboot. Which should be current, if chrony is truly syncing it.
On Sat, 2019-11-23 at 08:59 -0500, Sam Varshavchik wrote:
I have chrony running and pointing to my ISP's time server, not the Fedora pool, so it's always the same NTP server.
This box is up 24/7, and I just rebooted it. And I get this, after a reboot:
Nov 23 08:53:10 shorty chronyd[992]: System clock TAI offset set to 37 seconds Nov 23 08:53:10 shorty chronyd[992]: System clock wrong by -5.652285 seconds, adjustment started Nov 23 08:53:10 shorty chronyd[992]: System clock was stepped by -5.652285 seconds
This was a reboot, and not a shutdown.
Some PCs don't keep time well, and that can account for one PC being different from the others, regardless of CMOS battery condition.
When I was setting up a new server, I looked into the NTP versus chronyd thing, and came to the following conclusions.
NTP was better for computers that were continuously running. It keeps tinkering with the clock to keep it on time. And NTP has a long history, lots of things support it, and has well known behaviour.
Chronyd was better for computers are shutdown. It's designed to quickly reset your clock, but didn't continually tinker with it (that may be configurable, and I don't know what it's future behaviour will be). It's newer, and supposedly less widely supported.
Quick summary: https://medium.com/@codingmaths/centos-rhel-7-chronyd-v-s-ntp-service-5d6576...
On 2019-11-24 12:22, Sam Varshavchik wrote:
Louis Lagendijk writes:
On Sat, 2019-11-23 at 10:09 -0500, Sam Varshavchik wrote:
So, hwclock must be getting synced. But I don't see where hwclock would be getting called from. grepping /lib/systemd/system finds nothing. hwclock itself comes from util-linux, which doesn't install anything there.
Is chrony itself supposed to keep the hw clock synced? From https://chrony.tuxfamily.org/manual.html#rtcsync-directive::
4.2.55 rtcsync
The rtcsync directive enables a mode where the system time is periodically copied to the real time clock (RTC).
That looks like it. The rtcsync directive exist in my chrony.conf. I didn't put there, so it must be in the default chrony.conf
So, the hardware clock seems to be getting synced, but it loses some seconds during a reboot.
FWIW, this appears to be not working as expected on my system. The chrony.conf man page states:
On Linux, the RTC copy is performed by the kernel every 11 minutes.
This would indicate to me that within 11 minutes the clocks should be "sync'd" and be close.
I ran as script which collects the system time and the hwclock time. Just doing
date +%Y-%m-%d\ %H:%M:%S.%N%:z > tdiffs ; hwclock -r >> tdiffs
I then called it with 12 minute sleeps in the following manner
tdiff ; cat tdiffs ; sleep 720 ; tdiff ; cat tdiffs ; sleep 720 ;tdiff ; cat tdiffs
The results being
2019-11-24 13:15:34.760898924+08:00 2019-11-24 13:15:34.702288+08:00
2019-11-24 13:27:35.070572131+08:00 2019-11-24 13:27:34.999148+08:00
2019-11-24 13:39:36.083624152+08:00 2019-11-24 13:39:35.967988+08:00
As one can see, the difference continues to increase. If rtcsync were working I would expect the difference would remain fairly stable.
That it isn't working would seem to bare out the records of boots from earlier this year. When the system was up 24/7 from 4/14 to 4/22. And when the system was rebooted on 4/22 I see
Apr 22 18:25:21 meimei.greshko.com chronyd[850]: System clock wrong by 35.483104 seconds
Additionally, the difference between the 2nd and 3rd readings would suggest a an increase of .04421 seconds in a 12 minuted period or about 5 seconds/day. And that tracks with the 7 day period between boots.
On Sun, 24 Nov 2019 15:24:16 +1030 Tim via users wrote:
Some PCs don't keep time well, and that can account for one PC being different from the others, regardless of CMOS battery condition.
Amen :-). I've had at least one PC that would be off by about a minute every hour if I had relied on the motherboard clock.
On Sat, 2019-11-23 at 23:22 -0500, Sam Varshavchik wrote:
Louis Lagendijk writes:
On Sat, 2019-11-23 at 10:09 -0500, Sam Varshavchik wrote:
So, hwclock must be getting synced. But I don't see where hwclock would be getting called from. grepping /lib/systemd/system finds nothing. hwclock itself comes from util-linux, which doesn't install anything there.
Is chrony itself supposed to keep the hw clock synced? From https://chrony.tuxfamily.org/manual.html#rtcsync-directive::
4.2.55 rtcsync
The rtcsync directive enables a mode where the system time is periodically copied to the real time clock (RTC).
That looks like it. The rtcsync directive exist in my chrony.conf. I didn't put there, so it must be in the default chrony.conf
So, the hardware clock seems to be getting synced, but it loses some seconds during a reboot.
This could quite well be a failing battery for the RTC on your motherboard. Just replace it. It is mot likely a CR2032
-
Ed Greshko -
Louis Lagendijk -
Michael Thompson -
Sam Varshavchik -
Tim -
Tom Horsley