Here's ChatGPT feedback regarding your server logs...

Analyzing the provided log messages from a RHEL 9 Linux server, we can identify several events. Here's a breakdown of each event, including its probable cause and some examples:

1. ipa-dnskeysyncd LDAP Operations:
   - Messages from `ipa-dnskeysyncd` show LDAP operations related to DNSSEC key synchronization.
   - Example: `ipa-dnskeysyncd[75834]: ipaserver.dnssec.bindmgr: INFO attrs: <ldap.cidict.cidict object at 0x7f283b53f7f0>`.
   - Probable Cause: Routine operations of DNSSEC synchronization with IPA server.

2. DNS Notify Messages:
   - Messages from `named` indicate it received DNS notify messages for specific zones.
   - Example: `named[1463]: client @0x7fd61803ae68 10.12.112.3#49016: received notify for zone '0.0.0.0.0.0.a.0.0.8.9.2.4.0.6.2.ip6.arpa'`.
   - Probable Cause: Notification of changes in DNS zones, likely part of routine DNS zone updates.

3. dnssec-keyfromlabel Configuration Warnings:
   - Warnings from `dnssec-keyfromlabel` about missing configuration parameters.
   - Example: `dnssec-keyfromlabel[75984]: Configuration.cpp(96): Missing log.level in configuration. Using default value: INFO`.
   - Probable Cause: Default configuration being used due to missing specific configuration settings.

4. Systemd Service Messages:
   - `systemd` messages about `ipa-ods-exporter.service`.
   - Example: `systemd[1]: ipa-ods-exporter.service: Consumed 4.047s CPU time`.
   - Probable Cause: Normal systemd logging of service status.

5. Kernel Segmentation Fault (Core Dump):
   - Critical: A segmentation fault in `dnssec-keyfroml`.
   - Example: `kernel: dnssec-keyfroml[75991]: segfault at 18 ip 00007f7fba586284 sp 00007ffff3e834a8 error 4 in libsofthsm2.so[7f7fba4ef000+9b000]`.
   - Probable Cause: A crash in the `dnssec-keyfromlabel` process, possibly due to an issue with `libsofthsm2`.

6. Systemd Core Dump Processing:
   - Systemd handling of the core dump created by the crash.
   - Example: `systemd-coredump[75993]: Process 75991 (dnssec-keyfroml) of user 984 dumped core`.
   - Indicates that a core dump was generated for further analysis.

7. ipa-dnskeysyncd Python Traceback:
   - A Python traceback from `ipa-dnskeysyncd`, indicating an error in the script.
   - Example: `ipa-dnskeysyncd[75834]: ipapython.ipautil.CalledProcessError: Command [...] returned non-zero exit status -11`.
   - Probable Cause: The error seems related to the `dnssec-keyfromlabel` command failure.

8. Named TCP Receive Error:
   - Named server shutting down a TCP connection due to an error.
   - Example: `named[1463]: dispatch 0x7fd618009200: shutting down due to TCP receive error: connection reset`.
   - Probable Cause: Network-related issue or remote client reset the connection.

### Steps to Troubleshoot Core Dump

To investigate the core dump caused by `dnssec-keyfroml`:

1. Locate the Core Dump:
   - Core dumps are typically stored in `/var/lib/systemd/coredump/` or `/var/crash/` on RHEL systems.
   - Use `coredumpctl list` to list all core dumps and find the one related to `dnssec-keyfroml`.

2. Analyze the Core Dump:
   - Use `coredumpctl info <PID>` or `gdb` to analyze the core dump. Replace `<PID>` with the process ID of `dnssec-keyfroml`.
   - This will provide stack traces and other details about the state of the program at the time of the crash.

3. Check Dependencies:
   - Ensure that all dependencies, especially `libsofthsm2`, are correctly installed and not corrupted.

4. Review Configuration and Logs:
   - Check the configuration files for `dnssec-keyfromlabel` and related services.
   - Review system logs for any anomalies or errors leading up to the crash.

5. Test in a Controlled Environment:
   - If possible, replicate the issue in a test environment to better understand the conditions leading to the creation of the core dump file.

On Mon, Mar 4, 2024 at 10:54 PM Harry G Coin via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:
FYI.  El9, latest stable.


Mar 04 22:46:05 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: ipaserver.dnssec.bindmgr: INFO     attrs: <ldap.cidict.cidict object at 0x7f283b53f7f0>
Mar 04 22:46:05 registry1.1.quietfountain.com named[1463]: client @0x7fd61803ae68 10.12.112.3#49016: received notify for zone '0.0.0.0.0.0.a.0.0.8.9.2.4.0.6.2.ip6.arpa'
Mar 04 22:46:05 registry1.1.quietfountain.com named[1463]: client @0x7fd618191538 fc00:1002:c7::3#41560: received notify for zone '0.0.0.0.0.0.a.0.0.8.9.2.4.0.6.2.ip6.arpa'
Mar 04 22:46:05 registry1.1.quietfountain.com dnssec-keyfromlabel[75984]: Configuration.cpp(96): Missing log.level in configuration. Using default value: INFO
Mar 04 22:46:05 registry1.1.quietfountain.com dnssec-keyfromlabel[75984]: Configuration.cpp(96): Missing slots.mechanisms in configuration. Using default value: ALL
Mar 04 22:46:05 registry1.1.quietfountain.com dnssec-keyfromlabel[75984]: Configuration.cpp(124): Missing slots.removable in configuration. Using default value: false
Mar 04 22:46:05 registry1.1.quietfountain.com systemd[1]: ipa-ods-exporter.service: Deactivated successfully.
Mar 04 22:46:05 registry1.1.quietfountain.com systemd[1]: ipa-ods-exporter.service: Consumed 4.047s CPU time.
Mar 04 22:46:05 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: ipaserver.dnssec.bindmgr: INFO     attrs: <ldap.cidict.cidict object at 0x7f283b53faf0>
Mar 04 22:46:06 registry1.1.quietfountain.com dnssec-keyfromlabel[75991]: Configuration.cpp(96): Missing log.level in configuration. Using default value: INFO
Mar 04 22:46:06 registry1.1.quietfountain.com dnssec-keyfromlabel[75991]: Configuration.cpp(96): Missing slots.mechanisms in configuration. Using default value: ALL
Mar 04 22:46:06 registry1.1.quietfountain.com dnssec-keyfromlabel[75991]: Configuration.cpp(124): Missing slots.removable in configuration. Using default value: false
Mar 04 22:46:06 registry1.1.quietfountain.com named[1463]: client @0x7fd61803ae68 10.12.112.3#49016: received notify for zone '0.0.0.0.0.0.f.0.0.8.9.2.4.0.6.2.ip6.arpa'
Mar 04 22:46:06 registry1.1.quietfountain.com named[1463]: client @0x7fd618191538 fc00:1002:c7::3#41560: received notify for zone '0.0.0.0.0.0.f.0.0.8.9.2.4.0.6.2.ip6.arpa'
Mar 04 22:46:06 registry1.1.quietfountain.com kernel: dnssec-keyfroml[75991]: segfault at 18 ip 00007f7fba586284 sp 00007ffff3e834a8 error 4 in libsofthsm2.so[7f7fba4ef000+9b000] likely on CPU 0 (core 0, socket 0)
Mar 04 22:46:06 registry1.1.quietfountain.com kernel: Code: 75 f4 48 8b 53 30 49 89 4c 24 20 49 89 44 24 10 49 89 54 24 28 48 83 c4 08 4c 89 e0 5b 41 5c c3 66 0f 1f 44 00 00 f3 0f 1e fa <48> 8b 47 18 48 85 c0 74 44 4c 8d 47 10 4c 89 c7 eb 12 66 2e 0f 1f
Mar 04 22:46:06 registry1.1.quietfountain.com systemd[1]: Started Process Core Dump (PID 75992/UID 0).
Mar 04 22:46:06 registry1.1.quietfountain.com systemd-coredump[75993]: Resource limits disable core dumping for process 75991 (dnssec-keyfroml).
Mar 04 22:46:06 registry1.1.quietfountain.com systemd-coredump[75993]: [🡕] Process 75991 (dnssec-keyfroml) of user 984 dumped core.
Mar 04 22:46:06 registry1.1.quietfountain.com systemd[1]: systemd-coredump@40-75992-0.service: Deactivated successfully.
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: Traceback (most recent call last):
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]:   File "/usr/libexec/ipa/ipa-dnskeysyncd", line 130, in <module>
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]:     while ldap_connection.syncrepl_poll(all=1, msgid=ldap_search):
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]:   File "/usr/lib64/python3.9/site-packages/ldap/syncrepl.py", line 464, in syncrepl_poll
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]:     self.syncrepl_refreshdone()
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]:   File "/usr/lib/python3.9/site-packages/ipaserver/dnssec/keysyncer.py", line 128, in syncrepl_refreshdone
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]:     self.bindmgr.sync(self.dnssec_zones)
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]:   File "/usr/lib/python3.9/site-packages/ipaserver/dnssec/bindmgr.py", line 232, in sync
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]:     self.sync_zone(zone)
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]:   File "/usr/lib/python3.9/site-packages/ipaserver/dnssec/bindmgr.py", line 205, in sync_zone
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]:     self.install_key(zone, uuid, attrs, tempdir)
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]:   File "/usr/lib/python3.9/site-packages/ipaserver/dnssec/bindmgr.py", line 146, in install_key
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]:     result = ipautil.run(cmd, capture_output=True)
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]:   File "/usr/lib/python3.9/site-packages/ipapython/ipautil.py", line 599, in run
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]:     raise CalledProcessError(
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: ipapython.ipautil.CalledProcessError: CalledProcessError(Command ['/usr/sbin/dnssec-keyfromlabel', '-E', 'pkcs11', '-K', '/var/named/dyndb-ldap/ipa/master/_domainkey.rockstable.systems/XXXXXX', '-a', b'RSASHA256', '-l', b'pkcs11:object=YYYYY;pin-source=/var/lib/ipa/dnssec/softhsm_pin', '-P', b'20231101161909', '-A', 'none', '-I', 'none', '-D', 'none', '-E', 'pkcs11', '_domainkey.rockstable.systems.'] returned non-zero exit status -11: 'Found uninitialized token\nFound uninitialized token\nKey not found.\nPKCS11_load_public_key returned NULL\nFound uninitialized token\nKey not found.\nPKCS11_get_private_key returned NULL\ndnssec-keyfromlabel: warning: ENGINE_load_private_key failed (not found)\ndnssec-keyfromlabel: fatal: failed to get key _domainkey.rockstable.systems/RSASHA256: not found\n')
Mar 04 22:46:06 registry1.1.quietfountain.com named[1463]: dispatch 0x7fd618009200: shutting down due to TCP receive error: 2001:13c7:7002:3000::11#53: connection reset

--
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue