Time skew is 82 years off, with no replicas

[Kevin Konzem]

So in our environment, we had 3 freeipa 4.9.11 servers (ipa4, ipa5, ipa6) in a replicated
setup, DL1, and a couple weeks ago, something happened that broke the replication between
the three of them. We were able to get ipa6 up and running and accepting clients, so we
shut down ipa4 and ipa5 and removed all replication agreements, ruvs, and topology
segments related to ipa4 and ipa5. We are trying to clean up ipa6 before attempting to
stand up new replicas, and we are stuck on this one problem: when running ipa-healthcheck,
it complains that the time skew is over 24 hours, specifically it is 30145 days off, a bit
over 82 years.

ipa6 ~ $ ipa-healthcheck
[
  {
    "source": "ipahealthcheck.ds.dse",
    "check": "DSECheck",
    "result": "CRITICAL",
    "uuid": "ecc2c131-b86a-4851-a156-c304d77ebb0b",
    "when": "20230822183329Z",
    "duration": "0.012828",
    "kw": {
      "key": "DSSKEWLE0003",
      "items": [
        "Replication",
        "dc=DOMAIN,dc=DOMAIN,dc=DOMAIN",
        "Time Skew",
        "Skew: 30145 days, 2 hours, 20 minutes, 16 seconds"
      ],
      "msg": "The time skew is over 24 hours.  Setting
nsslapd-ignore-time-skew\nto \"on\" on each replica will allow replication to
continue, but if the\ntime skew continues to increase other serious replication problems
can\noccur."
    }

I was able to find mention of this method to correct time skew
issues:https://www.port389.org/docs/389ds/howto/howto-fix-and-reset-time-... , but
since we are now running on a single node with no replicas, I dont see how that would
help. How is it even possible to have a skew in time between replicas if there are no
replicas?

Thanks,
Kevin

[Kevin Konzem]

I did run the script to check the CSN generator states, this output is below:

./readNsState.py dse.ldif
nsState is HAAAAAAAAACdFOVkAAAAAAAAAAAAAAAAYBI+mwAAAAAEAAAAAAAAAA==
Little Endian
For replica cn=replica,cn=dc\3DDOMAIN\2Cdc\3DDOMAIN\2Cdc\3DDOMAIN,cn=mapping
tree,cn=config
  fmtstr=[H6x3QH6x]
  size=40
  len of nsstate is 40
  CSN generator state:
    Replica ID    : 28
    Sampled Time  : 1692734621
    Gen as csn    : 64e5149d000400280000
    Time as str   : Tue Aug 22 15:03:41 2023
    Local Offset  : 0
    Remote Offset : 2604536416
    Seq. num      : 4
    System time   : Tue Aug 22 15:04:27 2023
    Diff in sec.  : 46
    Day:sec diff  : 0:46

nsState is HQAAAAAAAADJBuVkAAAAAAAAAAAAAAAA4hcAAAAAAAABAAAAAAAAAA==
Little Endian
For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
  fmtstr=[H6x3QH6x]
  size=40
  len of nsstate is 40
  CSN generator state:
    Replica ID    : 29
    Sampled Time  : 1692731081
    Gen as csn    : 64e506c9000100290000
    Time as str   : Tue Aug 22 14:04:41 2023
    Local Offset  : 0
    Remote Offset : 6114
    Seq. num      : 1
    System time   : Tue Aug 22 15:04:27 2023
    Diff in sec.  : 3586
    Day:sec diff  : 0:3586

[Rob Crittenden]

Kevin Konzem via FreeIPA-users wrote:
>  I did run the script to check the CSN generator states, this output
> is below:
>  
>  ./readNsState.py dse.ldif
>  nsState is HAAAAAAAAACdFOVkAAAAAAAAAAAAAAAAYBI+mwAAAAAEAAAAAAAAAA==
>  Little Endian
>  For replica cn=replica,cn=dc\3DDOMAIN\2Cdc\3DDOMAIN\2Cdc\3DDOMAIN,cn=mapping
> tree,cn=config
>    fmtstr=[H6x3QH6x]
>    size=40
>    len of nsstate is 40
>    CSN generator state:
>      Replica ID    : 28
>      Sampled Time  : 1692734621
>      Gen as csn    : 64e5149d000400280000
>      Time as str   : Tue Aug 22 15:03:41 2023
>      Local Offset  : 0
>      Remote Offset : 2604536416
>      Seq. num      : 4
>      System time   : Tue Aug 22 15:04:27 2023
>      Diff in sec.  : 46
>      Day:sec diff  : 0:46
>  
>  nsState is HQAAAAAAAADJBuVkAAAAAAAAAAAAAAAA4hcAAAAAAAABAAAAAAAAAA==
>  Little Endian
>  For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>    fmtstr=[H6x3QH6x]
>    size=40
>    len of nsstate is 40
>    CSN generator state:
>      Replica ID    : 29
>      Sampled Time  : 1692731081
>      Gen as csn    : 64e506c9000100290000
>      Time as str   : Tue Aug 22 14:04:41 2023
>      Local Offset  : 0
>      Remote Offset : 6114
>      Seq. num      : 1
>      System time   : Tue Aug 22 15:04:27 2023
>      Diff in sec.  : 3586
>      Day:sec diff  : 0:3586 
Despite this being the only server left you still have replication
agreements remaining.

Try using ipa topology-segment or ipa-replica-manage to remove them.

rob

[Mark Reynolds]

On 8/24/23 11:46 AM, Rob Crittenden via FreeIPA-users wrote:
>  Kevin Konzem via FreeIPA-users wrote:
> > I did run the script to check the CSN generator states, this output is below:
> >
> > ./readNsState.py dse.ldif
> > nsState is HAAAAAAAAACdFOVkAAAAAAAAAAAAAAAAYBI+mwAAAAAEAAAAAAAAAA==
> > Little Endian
> > For replica cn=replica,cn=dc\3DDOMAIN\2Cdc\3DDOMAIN\2Cdc\3DDOMAIN,cn=mapping
> tree,cn=config
> >    fmtstr=[H6x3QH6x]
> >    size=40
> >    len of nsstate is 40
> >    CSN generator state:
> >      Replica ID    : 28
> >      Sampled Time  : 1692734621
> >      Gen as csn    : 64e5149d000400280000
> >      Time as str   : Tue Aug 22 15:03:41 2023
> >      Local Offset  : 0
> >      Remote Offset : 2604536416
> >      Seq. num      : 4
> >      System time   : Tue Aug 22 15:04:27 2023
> >      Diff in sec.  : 46
> >      Day:sec diff  : 0:46
> >
> > nsState is HQAAAAAAAADJBuVkAAAAAAAAAAAAAAAA4hcAAAAAAAABAAAAAAAAAA==
> > Little Endian
> > For replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> >    fmtstr=[H6x3QH6x]
> >    size=40
> >    len of nsstate is 40
> >    CSN generator state:
> >      Replica ID    : 29
> >      Sampled Time  : 1692731081
> >      Gen as csn    : 64e506c9000100290000
> >      Time as str   : Tue Aug 22 14:04:41 2023
> >      Local Offset  : 0
> >      Remote Offset : 6114
> >      Seq. num      : 1
> >      System time   : Tue Aug 22 15:04:27 2023
> >      Diff in sec.  : 3586
> >      Day:sec diff  : 0:3586
>  Despite this being the only server left you still have replication
>  agreements remaining.
>
>  Try using ipa topology-segment or ipa-replica-manage to remove them. 
Also, as long as replication is enabled on a database the CSN generation 
will still be generated.  As Rob said, these offsets are computed by 
remote replicas (through repl agreements).

Going back to the issue.  That is obviously a very high offset/skew.  We 
typically only see these "jumps" on AWS.  Was one of your replicas on 
AWS by any chance?  In the DS error log (on newer versions of 
389-ds-base) we log these jumps for debugging purposes.  Could you check 
your DS error log (/var/log/dirsrv/slapd-YOUR_INSTANCE/errors*) for 
"Detected large jump in CSN time", and please share that message if it 
exists.

Thanks,

Mark

>  rob
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- 
Directory Server Development Team