Rob,
That worked!
No errors on healthcheck now. getcert list shows the expected output as well (9
certificates tracked just like on the other two servers).
I am not sure how to look for the LDAP entry for the HTTP service on ipa1 to remove the
old certificate blob.
Thanks,
-Scott
-----Original Message-----
From: Rob Crittenden <rcritten(a)redhat.com>
Sent: Thursday, December 2, 2021 12:33 PM
To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
Cc: Dungan, Scott A. <sdungan(a)caltech.edu>
Subject: Re: [Freeipa-users] Re: ipa-healthcheck: ReplicationCheck ERROR
Try this. Move /var/lib/ipa/certs/httpd.crt to some place safe.
# getcert resubmit -i 20211130180109 -w -v
You'll be able to watch the progress. It should end up in MONITORING with a new
certificate.
Restart httpd.
So now you have a new, valid, known certificate for the service.
You'll need to modify the LDAP entry for this service to remove the original
certificate because it is unknown and likely to fail revocation by IPA tools as a result
(e.g. uninstallation). Figure out which of the userCertificate blobs is the old cert and
remove that value.
rob
Dungan, Scott A. via FreeIPA-users wrote:
So, that certificate appears to be the newly installed web cert,
signed by the internal ipa CA for the ipa1 server itself:
[root(a)ipa1.id.example.com ~]# getcert list -i 20211130180109 Number of
certificates and requests being tracked: 9.
Request ID '20211130180109':
status: MONITORING
stuck: no
key pair storage:
type=FILE,location='/var/lib/ipa/private/httpd.key',pinfile='/var/lib/ipa/passwds/ipa1.id.example.com-443-RSA'
certificate: type=FILE,location='/var/lib/ipa/certs/httpd.crt'
CA: IPA
issuer: CN=Certificate
Authority,O=ID.EXAMPLE.COM
subject: CN=
ipa1.id.example.com,O=ID.EXAMPLE.COM
expires: 2023-12-01 10:01:12 PST
dns:
ipa1.id.example.com,ipa-ca.id.example.com
principal name: HTTP/ipa1.id.example.com(a)ID.EXAMPLE.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
-Scott
-----Original Message-----
From: Rob Crittenden <rcritten(a)redhat.com>
Sent: Thursday, December 2, 2021 5:37 AM
To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
Cc: Dungan, Scott A. <sdungan(a)caltech.edu>
Subject: Re: [Freeipa-users] Re: ipa-healthcheck: ReplicationCheck
ERROR
So there was some data loss.
Re-issuing a cert generally involves revoking the existing one but since that isn't
there, this could be interesting.
What does the tracking for that cert look like? getcert list -i
20211130180109
rob
Dungan, Scott A. via FreeIPA-users wrote:
> Update: after 20-30 minutes, the ca replication errors stopped, leaving only the new
IPACertRevocation error and the 404 code:
>
> [root(a)ipa1.id.example.com ~]# ipa-healthcheck --failures-only ...
> ra.get_certificate(): Request failed with status 404: Non-2xx
> response from CA REST API: 404. Certificate ID 0x13 not found (404) [
> {
> "source": "ipahealthcheck.ipa.certs",
> "check": "IPACertRevocation",
> "result": "ERROR",
> "uuid": "c447d225-c712-4a77-b174-81f6ba008128",
> "when": "20211202025711Z",
> "duration": "3.825151",
> "kw": {
> "key": "20211130180109",
> "serial": 19,
> "error": "Certificate operation cannot be completed: Request
failed with status 404: Non-2xx response from CA REST API: 404. Certificate ID 0x13 not
found (404)",
> "msg": "Request for certificate serial number {serial} in
request {key} failed: {error}"
> }
> }
> ]
So there was some data loss.
Re-issuing a cert generally involves revoking the existing one but since that isn't
there, this could be interesting.
What does the tracking for that cert look like? getcert list -i
20211130180109
rob
>
> -----Original Message-----
> From: Dungan, Scott A. via FreeIPA-users
> <freeipa-users(a)lists.fedorahosted.org>
> Sent: Wednesday, December 1, 2021 5:18 PM
> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
> Cc: Dungan, Scott A. <sdungan(a)caltech.edu>
> Subject: [Freeipa-users] Re: ipa-healthcheck: ReplicationCheck ERROR
>
> Hi, Rob.
>
> I think I got ahead of myself and may have made things worse:
>
> Because servers iap2 and ipa3 are in agreement (ldif matches) and had no healthcheck
errors, and ipa1 did, I assumed that ipa1 needed to be re-initialized. The small number of
changes that would be lost on ipa1 was/is considered negligible. To re-initialize ipa1, I
ran the following:
>
> [root(a)ipa1.id.example.com ~]# ipa-csreplica-manage re-initialize --from
ipa3.id.example.com Directory Manager password:
>
> Update in progress, 7 seconds elapsed Update succeeded
>
> Looks good, but when I do a healthcheck again, now I get additional errors as well as
the original:
>
> [root(a)ipa1.id.example.com ~]# ipa-healthcheck --failures-only ...
> ra.get_certificate(): Request failed with status 404: Non-2xx response from CA REST
API: 404. Certificate ID 0x13 not found (404 [
> {
> "source": "ipahealthcheck.ds.replication",
> "check": "ReplicationCheck",
> "result": "ERROR",
> "uuid": "023f04e2-e8d8-459e-b2cb-bb516e30db07",
> "when": "20211202005925Z",
> "duration": "0.301820",
> "kw": {
> "key": "DSREPLLE0003",
> "items": [
> "Replication",
> "Agreement"
> ],
> "msg": "The replication agreement (
catoipa2.id.example.com)
under \"o=ipaca\" is not in synchronization.\nStatus messa: error (18) can't
acquire replica (incremental update transient warning. backing off, will retry update
later.)"
> }
> },
> {
> "source": "ipahealthcheck.ds.replication",
> "check": "ReplicationCheck",
> "result": "ERROR",
> "uuid": "6b7499ce-2138-415d-88ac-bf24f6f3a6db",
> "when": "20211202005925Z",
> "duration": "0.301851",
> "kw": {
> "key": "DSREPLLE0003",
> "items": [
> "Replication",
> "Agreement"
> ],
> "msg": "The replication agreement (
catoipa3.id.example.com)
under \"o=ipaca\" is not in synchronization.\nStatus messa: error (18) can't
acquire replica (incremental update transient warning. backing off, will retry update
later.)"
> }
> },
> {
> "source": "ipahealthcheck.ipa.certs",
> "check": "IPACertRevocation",
> "result": "ERROR",
> "uuid": "b02a8f85-985f-4b80-a099-fb3d415b2005",
> "when": "20211202005935Z",
> "duration": "2.663208",
> "kw": {
> "key": "20211130180109",
> "serial": 19,
> "error": "Certificate operation cannot be completed: Request
failed with status 404: Non-2xx response from CA REST API: 40 Certificate ID 0x13 not
found (404)",
> "msg": "Request for certificate serial number {serial} in
request {key} failed: {error}"
> }
> }
> ]
>
> -----Original Message-----
> From: Rob Crittenden <rcritten(a)redhat.com>
> Sent: Wednesday, December 1, 2021 11:50 AM
> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
> Cc: Dungan, Scott A. <sdungan(a)caltech.edu>
> Subject: Re: [Freeipa-users] ipa-healthcheck: ReplicationCheck ERROR
>
> Dungan, Scott A. via FreeIPA-users wrote:
>> We have 3 ipa servers, one of which is throwing an ERROR condition
>> during ipa-healthcheck for the "ReplicationCheck" test.
>> Ipa-healthcheck shows no errors when run from the other two replicas.
>> Looking back at the logs, it appears this started about ten days
>> ago, so it is not a transient issue as the output suggests:
>>
>>
>>
>> [root(a)ipa1.id.example.com]# ipa-healthcheck --failures-only
>>
>>
>>
>> [
>>
>> {
>>
>> "source": "ipahealthcheck.ds.replication",
>>
>> "check": "ReplicationCheck",
>>
>> "result": "ERROR",
>>
>> "uuid": "2b971ca3-678e-4c26-86a0-5b352027e7e8",
>>
>> "when": "20211201180013Z",
>>
>> "duration": "0.687812",
>>
>> "kw": {
>>
>> "key": "DSREPLLE0003",
>>
>> "items": [
>>
>> "Replication",
>>
>> "Agreement"
>>
>> ],
>>
>> "msg": "The replication agreement (
catoipa2.id.example.com)
>> under \"o=ipaca\" is not in synchronization.\nStatus message: error
>> (18) can't acquire replica (incremental update transient warning.
>> backing off, will retry update later.)"
>>
>> }
>>
>> },
>>
>> {
>>
>> "source": "ipahealthcheck.ds.replication",
>>
>> "check": "ReplicationCheck",
>>
>> "result": "ERROR",
>>
>> "uuid": "99436870-bc98-4ce8-84b1-c0b0806945c8",
>>
>> "when": "20211201180013Z",
>>
>> "duration": "0.687829",
>>
>> "kw": {
>>
>> "key": "DSREPLLE0003",
>>
>> "items": [
>>
>> "Replication",
>>
>> "Agreement"
>>
>> ],
>>
>> "msg": "The replication agreement (
catoipa3.id.example.com)
>> under \"o=ipaca\" is not in synchronization.\nStatus message: error
>> (18) can't acquire replica (incremental update transient warning.
>> backing off, will retry update later.)"
>>
>> }
>>
>> }
>>
>> ]
>>
>>
>>
>> 389-ds error logs show a slew of these:
>>
>>
>>
>> [30/Nov/2021:23:41:35.277399980 -0800] - ERR - NSMMReplicationPlugin
>> - send_updates - agmt="cn=caToipa3.id.example.com" (ipa2:389):
>> Missing data encountered. If the error persists the replica must be
reinitialized.
>>
>> [30/Nov/2021:23:41:38.288003253 -0800] - ERR -
>> agmt="cn=caToipa3.id.example.com" (ipa3:389) - clcache_load_buffer -
>> Can't locate CSN 6197e149000000060000 in the changelog (DB rc=-30988).
>> If replication stops, the consumer may need to be reinitialized.
>>
>> [30/Nov/2021:23:41:38.289713999 -0800] - ERR - NSMMReplicationPlugin
>> - send_updates - agmt="cn=caToipa3.id.example.com" (ipa3:389):
>> Missing data encountered. If the error persists the replica must be
reinitialized.
>>
>>
>>
>> That would seem to suggest running a "ipa-replica-manage
>> re-initialize --from $SERVER_TO_PULL_FROM" may resolve the issue,
>> but before we try that, is there anything else we should look at?
>
> You use ipa-csreplica-manage to manage the CA replication agreements.
> But yes, it looks like you need to re-initialize some of them.
>
> I'd suggest dump the ldif of the two to be re-inited and see if there are any
entries there not recorded in the one you will re-init from to see if there is any
potential data loss.
>
> rob
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to
> freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct:
>
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
>
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedo
> r
ahosted.org Do not reply to spam on the list, report it:
>
https://pagure.io/fedora-infrastructure
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to
> freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct:
>
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
>
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedo
> r
ahosted.org Do not reply to spam on the list, report it:
>
https://pagure.io/fedora-infrastructure
>
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedor
ahosted.org Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure