What's the 'current best practice' for what you might call a 'fully deployed' freeipa install (meaning one that uses DNSSEC and all the documented capability subsections)?
From what I can tell, there are two approaches:
Approach 1: Run it in a VM, then from time to time shut it down, snapshot the 'known good' image, then bring the VM back up'. When there's a problem that's not 'trivial', just revert to the 'known good' snapshot and pay the price of re-entry of lost data.
The reasoning here is: Freeipa is much too interwoven among all the sub-packages and dependency-hell, the time required to re-enter data necessary to bring a backup current is at least known, compared to the almost impossible-to-guess time and multi-package expertise required to diagnosis much less cure problems in anything like the time available in production environments.
Approach 2: Use the ipa-backup, ipa-restore commands. Just keep a fresh install of the unconfigured pacakges, do ipa-restore and it will get every single detail correct across all the available documented functionality, databases, including the dnssec ones and the other 'trust' addons, etc.
The reasoning here is: It's better than approach 1 in that you don't have to take down the system even for a moment to get a 'known good' snapshot. However, you have to trust there is testing of 'everything the ipa-backup ipa-restore' process does actually does test the whole of the documented functionality.
For production reasons we'd prefer approach 2, as taking the system down and bringing it back up as in approach 1 introduces a momentary loss of capability but has the benefit of 'total confidence'. But is it really 'the best approach' in the production world?
Thanks!
Harry G Coin via FreeIPA-users wrote:
What's the 'current best practice' for what you might call a 'fully deployed' freeipa install (meaning one that uses DNSSEC and all the documented capability subsections)?
From what I can tell, there are two approaches:
Approach 1: Run it in a VM, then from time to time shut it down, snapshot the 'known good' image, then bring the VM back up'. When there's a problem that's not 'trivial', just revert to the 'known good' snapshot and pay the price of re-entry of lost data.
The reasoning here is: Freeipa is much too interwoven among all the sub-packages and dependency-hell, the time required to re-enter data necessary to bring a backup current is at least known, compared to the almost impossible-to-guess time and multi-package expertise required to diagnosis much less cure problems in anything like the time available in production environments.
Yes, this can help back out of failed upgrades, for example. Note that this and any backup and restore procedure can wreak havoc on replication. If you effectively take one server back in time then you should re-init it from another server.
I'm not sure why you bring up dependencies in the context of backup and restore.
Approach 2: Use the ipa-backup, ipa-restore commands. Just keep a fresh install of the unconfigured pacakges, do ipa-restore and it will get every single detail correct across all the available documented functionality, databases, including the dnssec ones and the other 'trust' addons, etc.
The reasoning here is: It's better than approach 1 in that you don't have to take down the system even for a moment to get a 'known good' snapshot. However, you have to trust there is testing of 'everything the ipa-backup ipa-restore' process does actually does test the whole of the documented functionality.
For production reasons we'd prefer approach 2, as taking the system down and bringing it back up as in approach 1 introduces a momentary loss of capability but has the benefit of 'total confidence'. But is it really 'the best approach' in the production world?
Backups are a good thing to have, but restoration should be due to catastrophe. When doing a restore recommend deleting all other servers and re-installing. You could conceivably re-init from the restored server as well, but it really is a nuke from orbit operation.
In either situation there will be down time.
Some level of backup is essentially online with MMR in 389-ds. This can't prevent undesired loss or modification of data but it otherwise provides the uptime you seem to be looking for.
rob
Thanks Rob
Replies to questions interposed below.
On 10/17/23 11:53, Rob Crittenden wrote:
Harry G Coin via FreeIPA-users wrote:
What's the 'current best practice' for what you might call a 'fully deployed' freeipa install (meaning one that uses DNSSEC and all the documented capability subsections)?
From what I can tell, there are two approaches:
Approach 1: Run it in a VM, then from time to time shut it down, snapshot the 'known good' image, then bring the VM back up'. When there's a problem that's not 'trivial', just revert to the 'known good' snapshot and pay the price of re-entry of lost data.
The reasoning here is: Freeipa is much too interwoven among all the sub-packages and dependency-hell, the time required to re-enter data necessary to bring a backup current is at least known, compared to the almost impossible-to-guess time and multi-package expertise required to diagnosis much less cure problems in anything like the time available in production environments.
Yes, this can help back out of failed upgrades, for example. Note that this and any backup and restore procedure can wreak havoc on replication. If you effectively take one server back in time then you should re-init it from another server.
Could you flesh out 'the official' command sequence for that? For example:
1. Restore the VM image to a 'known good' image (known-good means ipa-healthcheck reports no errors).
2. Boot the VM.
3. If there is no replication involved, re-enter data lost during the downtime and you're done. Otherwise:
4. If there is replication involved, especially multi-master replication: then run these commands in this order on these systems: <Rob will fill this in? Please?>
I'm not sure why you bring up dependencies in the context of backup and restore.
'security' and 'other' seemingly 'unrelated' 'upgrades' to packages n levels deep but whose previously un-noticed freeipa killing race-condition or other bug manifests after the upgrade. I find myself obligated to prevent any security or other change from happening until the lowest possible usage times. For example today's 'random freeipa bother' is:
Problem: cannot install both protobuf-3.5.0-15.el8.x86_64 and protobuf-3.19.0-2.el8s.x86_64 - package liborc1-1.7.9-1.el8.x86_64 requires libprotobuf.so.15()(64bit), but none of the providers can be installed - cannot install the best update candidate for package protobuf-3.19.0-2.el8s.x86_64 - cannot install the best update candidate for package liborc1-1.7.5-1.el8s.x86_64 (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
Also, there's a bit of a blurring of what comes to mind when the words 'multi-master' are adverstised, then the 'reality hair' (or at least AFAIK) in the kasp db and perhaps other places (SOA), where one system 'is more master than others even with m-m agreements'. There is 'lore' to do with 'manual steps' to 'move' 'special' files among what are otherwise thought of as 'master' systems. This is not a complaint, I'm glad there is at least some way, but it generates worry there are other sub-systems that also require 'as yet unlearned lore' to actually provide 'a whole system' from a backup.
Approach 2: Use the ipa-backup, ipa-restore commands. Just keep a fresh install of the unconfigured pacakges, do ipa-restore and it will get every single detail correct across all the available documented functionality, databases, including the dnssec ones and the other 'trust' addons, etc.
The reasoning here is: It's better than approach 1 in that you don't have to take down the system even for a moment to get a 'known good' snapshot. However, you have to trust there is testing of 'everything the ipa-backup ipa-restore' process does actually does test the whole of the documented functionality.
For production reasons we'd prefer approach 2, as taking the system down and bringing it back up as in approach 1 introduces a momentary loss of capability but has the benefit of 'total confidence'. But is it really 'the best approach' in the production world?
Backups are a good thing to have, but restoration should be due to catastrophe. When doing a restore recommend deleting all other servers and re-installing. You could conceivably re-init from the restored server as well, but it really is a nuke from orbit operation.
This is really helpful, thank you. The point then of 'ipa-backup' is-- an offline file meant to be used if everything burns to the ground, not to recover this or that failed master or replica.
In either situation there will be down time.
I agree. What I was really hoping for was a way to not take a 'known good' system down in order to get a 'known-everything-synced' snapshot. While filesystems (like xfs, and others) have means to 'freeze' or 'snapshot', as far as I know freeipa doesn't have a way to 'flush write caches and pause pending writes across all subsystems'.
Some level of backup is essentially online with MMR in 389-ds. This can't prevent undesired loss or modification of data but it otherwise provides the uptime you seem to be looking for.
All about uptime! Thanks Rob
Harry
rob
On 17/10/2023 19.32, Harry G Coin via FreeIPA-users wrote:
'security' and 'other' seemingly 'unrelated' 'upgrades' to packages n levels deep but whose previously un-noticed freeipa killing race-condition or other bug manifests after the upgrade. I find myself obligated to prevent any security or other change from happening until the lowest possible usage times. For example today's 'random freeipa bother' is:
Problem: cannot install both protobuf-3.5.0-15.el8.x86_64 and protobuf-3.19.0-2.el8s.x86_64 - package liborc1-1.7.9-1.el8.x86_64 requires libprotobuf.so.15()(64bit), but none of the providers can be installed - cannot install the best update candidate for package protobuf-3.19.0-2.el8s.x86_64 - cannot install the best update candidate for package liborc1-1.7.5-1.el8s.x86_64 (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
How did you end up with Hadoop-related libraries on your IPA server? Did you install additional services and EPEL on your IPA server?
Christian
Hi,
this guide explains the possible strategies for disaster recovery: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm...
And that one how to recover: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm...
Hope this helps, flo
On Tue, Oct 17, 2023 at 7:50 PM Christian Heimes via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
On 17/10/2023 19.32, Harry G Coin via FreeIPA-users wrote:
'security' and 'other' seemingly 'unrelated' 'upgrades' to packages n levels deep but whose previously un-noticed freeipa killing race-condition or other bug manifests after the upgrade. I find myself obligated to prevent any security or other change from happening until the lowest possible usage times. For example today's 'random freeipa bother' is:
Problem: cannot install both protobuf-3.5.0-15.el8.x86_64 and protobuf-3.19.0-2.el8s.x86_64
- package liborc1-1.7.9-1.el8.x86_64 requires libprotobuf.so.15()(64bit),
but none of the providers can be installed
- cannot install the best update candidate for package
protobuf-3.19.0-2.el8s.x86_64
- cannot install the best update candidate for package
liborc1-1.7.5-1.el8s.x86_64 (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
How did you end up with Hadoop-related libraries on your IPA server? Did you install additional services and EPEL on your IPA server?
Christian
-- Christian Heimes Principal Software Engineer, Identity Management and Platform Security
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Tue, Oct 17, 2023 at 7:50 PM Christian Heimes via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
On 17/10/2023 19.32, Harry G Coin via FreeIPA-users wrote:'security' and 'other' seemingly 'unrelated' 'upgrades' to packages n levels deep but whose previously un-noticed freeipa killing race-condition or other bug manifests after the upgrade. I find myself obligated to prevent any security or other change from happening until the lowest possible usage times. For example today's 'random freeipa bother' is: Problem: cannot install both protobuf-3.5.0-15.el8.x86_64 and protobuf-3.19.0-2.el8s.x86_64 - package liborc1-1.7.9-1.el8.x86_64 requires libprotobuf.so.15()(64bit), but none of the providers can be installed - cannot install the best update candidate for package protobuf-3.19.0-2.el8s.x86_64 - cannot install the best update candidate for package liborc1-1.7.5-1.el8s.x86_64 (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)How did you end up with Hadoop-related libraries on your IPA server? Did you install additional services and EPEL on your IPA server?
To gain access to the file system published by the multi-rack high availability file system at https://ceph.io, named 'cephfs' (a native fs akin to nfs in some ways) one must install ceph-common. That package comes one per version of major ceph releases. That appears to play badly with freeipa packaging. I was hoping by waiting patiently the packagers would figure that out for us. Dependency hell strikes again.
Harry
On 18/10/2023 16.57, Harry G Coin wrote:
On Tue, Oct 17, 2023 at 7:50 PM Christian Heimes via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
On 17/10/2023 19.32, Harry G Coin via FreeIPA-users wrote:'security' and 'other' seemingly 'unrelated' 'upgrades' to packages n levels deep but whose previously un-noticed freeipa killing race-condition or other bug manifests after the upgrade. I find myself obligated to prevent any security or other change from happening until the lowest possible usage times. For example today's 'random freeipa bother' is: Problem: cannot install both protobuf-3.5.0-15.el8.x86_64 and protobuf-3.19.0-2.el8s.x86_64 - package liborc1-1.7.9-1.el8.x86_64 requires libprotobuf.so.15()(64bit), but none of the providers can be installed - cannot install the best update candidate for package protobuf-3.19.0-2.el8s.x86_64 - cannot install the best update candidate for package liborc1-1.7.5-1.el8s.x86_64 (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)How did you end up with Hadoop-related libraries on your IPA server? Did you install additional services and EPEL on your IPA server?To gain access to the file system published by the multi-rack high availability file system at https://ceph.io, named 'cephfs' (a native fs akin to nfs in some ways) one must install ceph-common. That package comes one per version of major ceph releases. That appears to play badly with freeipa packaging. I was hoping by waiting patiently the packagers would figure that out for us. Dependency hell strikes again.
You are living a dangerous life, you are running an untested and unsupported configuration of FreeIPA. All our docs *strongly* advise against additional services on an IPA server, e.g. https://www.freeipa.org/page/Deployment_Recommendations#freeipa-server-exclu... . Third party repositories with conflicting packages are even more problematic.
On 10/18/23 10:33, Christian Heimes wrote:
On 18/10/2023 16.57, Harry G Coin wrote:
On Tue, Oct 17, 2023 at 7:50 PM Christian Heimes via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
On 17/10/2023 19.32, Harry G Coin via FreeIPA-users wrote:'security' and 'other' seemingly 'unrelated' 'upgrades' to packages n levels deep but whose previously un-noticed freeipa killing race-condition or other bug manifests after the upgrade. I find myself obligated to prevent any security or other change from happening until the lowest possible usage times. For example today's 'random freeipa bother' is: Problem: cannot install both protobuf-3.5.0-15.el8.x86_64 and protobuf-3.19.0-2.el8s.x86_64 - package liborc1-1.7.9-1.el8.x86_64 requires libprotobuf.so.15()(64bit), but none of the providers can be installed - cannot install the best update candidate for package protobuf-3.19.0-2.el8s.x86_64 - cannot install the best update candidate for package liborc1-1.7.5-1.el8s.x86_64 (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)How did you end up with Hadoop-related libraries on your IPA server? Did you install additional services and EPEL on your IPA server?To gain access to the file system published by the multi-rack high availability file system at https://ceph.io, named 'cephfs' (a native fs akin to nfs in some ways) one must install ceph-common. That package comes one per version of major ceph releases. That appears to play badly with freeipa packaging. I was hoping by waiting patiently the packagers would figure that out for us. Dependency hell strikes again.
You are living a dangerous life, you are running an untested and unsupported configuration of FreeIPA. All our docs *strongly* advise against additional services on an IPA server, e.g. https://www.freeipa.org/page/Deployment_Recommendations#freeipa-server-exclu... . Third party repositories with conflicting packages are even more problematic.
Thanks Christian. Might you publish a list of all the packages in the repos that can't be installed on a freeipa box? Can a freeipa system be an NFS client? Which file systems used by multiple tens of thousands around the world should avoid freeipa?
-- Christian Heimes Principal Software Engineer, Identity Management and Platform Security
Red Hat GmbH,https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
Harry G Coin via FreeIPA-users wrote:
On 10/18/23 10:33, Christian Heimes wrote:
On 18/10/2023 16.57, Harry G Coin wrote:
On Tue, Oct 17, 2023 at 7:50 PM Christian Heimes via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
On 17/10/2023 19.32, Harry G Coin via FreeIPA-users wrote:'security' and 'other' seemingly 'unrelated' 'upgrades' to packages n levels deep but whose previously un-noticed freeipa killing race-condition or other bug manifests after the upgrade. I find myself obligated to prevent any security or other change from happening until the lowest possible usage times. For example today's 'random freeipa bother' is: Problem: cannot install both protobuf-3.5.0-15.el8.x86_64 and protobuf-3.19.0-2.el8s.x86_64 - package liborc1-1.7.9-1.el8.x86_64 requires libprotobuf.so.15()(64bit), but none of the providers can be installed - cannot install the best update candidate for package protobuf-3.19.0-2.el8s.x86_64 - cannot install the best update candidate for package liborc1-1.7.5-1.el8s.x86_64 (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)How did you end up with Hadoop-related libraries on your IPA server? Did you install additional services and EPEL on your IPA server?To gain access to the file system published by the multi-rack high availability file system at https://ceph.io, named 'cephfs' (a native fs akin to nfs in some ways) one must install ceph-common. That package comes one per version of major ceph releases. That appears to play badly with freeipa packaging. I was hoping by waiting patiently the packagers would figure that out for us. Dependency hell strikes again.
You are living a dangerous life, you are running an untested and unsupported configuration of FreeIPA. All our docs *strongly* advise against additional services on an IPA server, e.g. https://www.freeipa.org/page/Deployment_Recommendations#freeipa-server-exclu... . Third party repositories with conflicting packages are even more problematic.
Thanks Christian. Might you publish a list of all the packages in the repos that can't be installed on a freeipa box? Can a freeipa system be an NFS client? Which file systems used by multiple tens of thousands around the world should avoid freeipa?
In this case it looks like a repo problem, not "rpm hell". It's completely unrelated to IPA. These are not IPA dependencies.
IPA connects a lot of disparate services together into a whole. There are only so many combinations we can test. This is why we recommend keeping things vanilla.
That is the point he was trying to make.
rob
On 10/18/23 15:55, Rob Crittenden wrote:
Harry G Coin via FreeIPA-users wrote:
On 10/18/23 10:33, Christian Heimes wrote:
On 18/10/2023 16.57, Harry G Coin wrote:
On Tue, Oct 17, 2023 at 7:50 PM Christian Heimes via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
On 17/10/2023 19.32, Harry G Coin via FreeIPA-users wrote:'security' and 'other' seemingly 'unrelated' 'upgrades' to packages n levels deep but whose previously un-noticed freeipa killing race-condition or other bug manifests after the upgrade. I find myself obligated to prevent any security or other change from happening until the lowest possible usage times. For example today's 'random freeipa bother' is: Problem: cannot install both protobuf-3.5.0-15.el8.x86_64 and protobuf-3.19.0-2.el8s.x86_64 - package liborc1-1.7.9-1.el8.x86_64 requires libprotobuf.so.15()(64bit), but none of the providers can be installed - cannot install the best update candidate for package protobuf-3.19.0-2.el8s.x86_64 - cannot install the best update candidate for package liborc1-1.7.5-1.el8s.x86_64 (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)How did you end up with Hadoop-related libraries on your IPA server? Did you install additional services and EPEL on your IPA server?To gain access to the file system published by the multi-rack high availability file system at https://ceph.io, named 'cephfs' (a native fs akin to nfs in some ways) one must install ceph-common. That package comes one per version of major ceph releases. That appears to play badly with freeipa packaging. I was hoping by waiting patiently the packagers would figure that out for us. Dependency hell strikes again.
You are living a dangerous life, you are running an untested and unsupported configuration of FreeIPA. All our docs *strongly* advise against additional services on an IPA server, e.g. https://www.freeipa.org/page/Deployment_Recommendations#freeipa-server-exclu... . Third party repositories with conflicting packages are even more problematic.
Thanks Christian. Might you publish a list of all the packages in the repos that can't be installed on a freeipa box? Can a freeipa system be an NFS client? Which file systems used by multiple tens of thousands around the world should avoid freeipa?
In this case it looks like a repo problem, not "rpm hell". It's completely unrelated to IPA. These are not IPA dependencies.
IPA connects a lot of disparate services together into a whole. There are only so many combinations we can test. This is why we recommend keeping things vanilla.
That is the point he was trying to make.
rob
Thanks Rob and Christian, Perhaps you'd agree it's necessary for freeipa to function as a client to the most used file systems (not server, which perhaps what Christian was pointing at). In the case of ceph, the client package is ceph-common. I can see how running a samba server and other file system servers would be problematic. But the need to be able to operate with network file system clients (particularly a kernel level file system) seems a necessary one.
Harry G Coin wrote:
On 10/18/23 15:55, Rob Crittenden wrote:
Harry G Coin via FreeIPA-users wrote:
On 10/18/23 10:33, Christian Heimes wrote:
On 18/10/2023 16.57, Harry G Coin wrote:
On Tue, Oct 17, 2023 at 7:50 PM Christian Heimes via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
On 17/10/2023 19.32, Harry G Coin via FreeIPA-users wrote: > 'security' and 'other' seemingly 'unrelated' 'upgrades' to > packages n levels deep but whose previously un-noticed freeipa > killing race-condition or other bug manifests after the > upgrade. I find myself obligated to prevent any security or > other change from happening until the lowest possible usage > times. For example today's 'random freeipa bother' is: > > Problem: cannot install both protobuf-3.5.0-15.el8.x86_64 and > protobuf-3.19.0-2.el8s.x86_64 > - package liborc1-1.7.9-1.el8.x86_64 requires > libprotobuf.so.15()(64bit), but none of the providers can be > installed > - cannot install the best update candidate for package > protobuf-3.19.0-2.el8s.x86_64 > - cannot install the best update candidate for package > liborc1-1.7.5-1.el8s.x86_64 > (try to add '--allowerasing' to command line to replace > conflicting packages or '--skip-broken' to skip uninstallable > packages or '--nobest' to use not only best candidate packages) > How did you end up with Hadoop-related libraries on your IPA server? Did you install additional services and EPEL on your IPA server?
To gain access to the file system published by the multi-rack high availability file system at https://ceph.io, named 'cephfs' (a native fs akin to nfs in some ways) one must install ceph-common. That package comes one per version of major ceph releases. That appears to play badly with freeipa packaging. I was hoping by waiting patiently the packagers would figure that out for us. Dependency hell strikes again.
You are living a dangerous life, you are running an untested and unsupported configuration of FreeIPA. All our docs *strongly* advise against additional services on an IPA server, e.g. https://www.freeipa.org/page/Deployment_Recommendations#freeipa-server-exclu...
. Third party repositories with conflicting packages are even more problematic.
Thanks Christian. Might you publish a list of all the packages in the repos that can't be installed on a freeipa box? Can a freeipa system be an NFS client? Which file systems used by multiple tens of thousands around the world should avoid freeipa?
In this case it looks like a repo problem, not "rpm hell". It's completely unrelated to IPA. These are not IPA dependencies.
IPA connects a lot of disparate services together into a whole. There are only so many combinations we can test. This is why we recommend keeping things vanilla.
That is the point he was trying to make.
rob
Thanks Rob and Christian, Perhaps you'd agree it's necessary for freeipa to function as a client to the most used file systems (not server, which perhaps what Christian was pointing at). In the case of ceph, the client package is ceph-common. I can see how running a samba server and other file system servers would be problematic. But the need to be able to operate with network file system clients (particularly a kernel level file system) seems a necessary one.
Yes. As I said, it's a matter of what we can test. We test NFS mounts but not ceph.
When users report problems with untested software, we tend to point our finger that way.
Again, in your case it's a repository issue, not a software issue.
rob
On 10/18/23 10:33, Christian Heimes wrote:
On 18/10/2023 16.57, Harry G Coin wrote:
On Tue, Oct 17, 2023 at 7:50 PM Christian Heimes via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
On 17/10/2023 19.32, Harry G Coin via FreeIPA-users wrote:'security' and 'other' seemingly 'unrelated' 'upgrades' to packages n levels deep but whose previously un-noticed freeipa killing race-condition or other bug manifests after the upgrade. I find myself obligated to prevent any security or other change from happening until the lowest possible usage times. For example today's 'random freeipa bother' is: Problem: cannot install both protobuf-3.5.0-15.el8.x86_64 and protobuf-3.19.0-2.el8s.x86_64 - package liborc1-1.7.9-1.el8.x86_64 requires libprotobuf.so.15()(64bit), but none of the providers can be installed - cannot install the best update candidate for package protobuf-3.19.0-2.el8s.x86_64 - cannot install the best update candidate for package liborc1-1.7.5-1.el8s.x86_64 (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)How did you end up with Hadoop-related libraries on your IPA server? Did you install additional services and EPEL on your IPA server?To gain access to the file system published by the multi-rack high availability file system at https://ceph.io, named 'cephfs' (a native fs akin to nfs in some ways) one must install ceph-common. That package comes one per version of major ceph releases. That appears to play badly with freeipa packaging. I was hoping by waiting patiently the packagers would figure that out for us. Dependency hell strikes again.
You are living a dangerous life, you are running an untested and unsupported configuration of FreeIPA. All our docs *strongly* advise against additional services on an IPA server, e.g. https://www.freeipa.org/page/Deployment_Recommendations#freeipa-server-exclu... . Third party repositories with conflicting packages are even more problematic.
Update: I think you'd be surprised to learn what was necessary to 'fix' the above problem and restore normal operations. Just these two commands:
dnf remove ceph-common
dnf install ceph-common
... you'd think RHEL/yum/dnf would be able to figure that out on it's own with just the 'dnf upgrade' -- but not so much.
HC
On 10/18/23 07:30, Florence Blanc-Renaud via FreeIPA-users wrote:
Hi,
this guide explains the possible strategies for disaster recovery: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm...
And that one how to recover: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm...
Hope this helps, flo
Thanks Flo!
Fundamentally freeipa is having a tension in that restoration from system loss is almost never 'a committee or team effort' but a one person task--- but freeipa now has such richness that few people have the bandwidth to know all the necessary details for each of the distinct complex subdomains freeipa integrates. A goal yet to be attained is a freeipa 'best practice' advice that includes all the details necessary to restore prior functionality ( snapshot the vm? ipa-backup? the CA thing? Kasp.db? Subdomain account number ranges on replicas? Does promoting a replica fix the SOA record? Is when 'a database that's too old' a testable fact or like when your coffee temp is just right? ) . Freeipa ought to have within itself such maps as necessary to itself comprehend the intended role of the system being restored then maneuver and marshal such file changes 'from what it used to be' to 'what it needs to be' to make the system fill its intended role -- without requiring the system-restore person to be expert in each of freeipa's 'deep dive expertise file areas'.
The rest of this post is detail on the above, and probably only interesting for few.
In general all the above advice flo linked to includes most, but not all required to actually accomplish restoration of the functional experience prior to the loss. Perhaps the advice should be qualified to apply only when the supplemental packages to do with trust and dns are not in use. You might consider advising folks to use ipa-healthcheck for the purpose of deciding 'how bad the current situation is' on this or that system. Too is missing the advice relating to version management among 'masters' and 'replicas' (the order in which upgrades get installed among systems) and whether backups made by 'the last one' remain usable by 'the next one'.
Notice that the detail in the first link provided there makes the blanket statement affirming it is possible to replace a lost server by creating a new replica. However, did the author take responsibility for all of freeipa's parts when that was written? For example if the server holding the file kasp.db fails, does that statement yet apply? Is there a comprehensive list of 'files that are exceptions to the rule' maintained by anyone (sub account number ranges on replicas)?
The next paragraph mentions VM snapshots, but leaves it to the reader to guess when 'a too outdated database' (4.2 sec 6) is or isn't the case. Too, does not mention that's not an answer if replication is active (timing issues, the documentation is missing about whether and when to walk users through the 're-init' Rob mentioned was required upstream). Might freeipa's replication setup itself notice when 'systemctl stop sssd;sss_cache -e;systemctl start sssd' is necessary and do that itself?
The last paragraph mentions idM backups, detailed in section 5, which too I think might require a list of important files not backed up, and might too include Rob's advise that such is only useful when 'everything has burned to the ground and a fresh install on all systems is the next step'.
Section 3.2 might make more clear freeipa does not just have two states, 'master' and the term rhel uses 'regular replica' relevant to integrity, but at least six major variants relevant to integrity: replica without CA and not dnssec master, replica with ca but not dnssec master, etc. .. and 'master with ca and dnssec db', etc. It's likely this list may double in length in the case of KRA usage. And potentially need to be entirely revised in light of sub-domain account number range allocation. Also the freeipa instance mentioned in SOA records has a status of its own that needs thought in the 'promoting replica to master' advice.
Does the advice of 'hidden replica' saving the day cover such files on the same list as kasp.db? If it's a manual step required as presently documented then ....?
Last: As freeipa is itself in the 'policy' services sector, is not what ansible provides something the internal maps freeipa maintains a better place to capture and deploy that knowledge?
Thanks all!
Harry Coin
Harry G Coin via FreeIPA-users wrote:
On 10/18/23 07:30, Florence Blanc-Renaud via FreeIPA-users wrote:
Hi,
this guide explains the possible strategies for disaster recovery: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm...
And that one how to recover: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm...
Hope this helps, flo
Thanks Flo!
Fundamentally freeipa is having a tension in that restoration from system loss is almost never 'a committee or team effort' but a one person task--- but freeipa now has such richness that few people have the bandwidth to know all the necessary details for each of the distinct complex subdomains freeipa integrates. A goal yet to be attained is a freeipa 'best practice' advice that includes all the details necessary to restore prior functionality ( snapshot the vm? ipa-backup? the CA thing? Kasp.db? Subdomain account number ranges on replicas? Does promoting a replica fix the SOA record? Is when 'a database that's too old' a testable fact or like when your coffee temp is just right? ) . Freeipa ought to have within itself such maps as necessary to itself comprehend the intended role of the system being restored then maneuver and marshal such file changes 'from what it used to be' to 'what it needs to be' to make the system fill its intended role -- without requiring the system-restore person to be expert in each of freeipa's 'deep dive expertise file areas'.
This was quite the dense set of questions :-). I'll try to cover everything.
As far as I'm aware, ipa-backup and ipa-restore save and restore all things necessary to recover an IPA installation. This does hedge a bit. If, for example, you back up a server that doesn't have optional services installed, you could be in trouble. The CA, for example. The user has some responsibility in identifying the most critical server(s) to back up.
The DNSSEC keys are stored in LDAP so can be recovered, if missing, AFAIK. DNS isn't my area. But we do back up the kasp.db in any case, if DNS is configured.
I'm not sure what you mean regarding SOA. It is...ahem...problematic in that all servers may have different ones, but this is due to not having an atomic way to increment values over replication.
The rest of this post is detail on the above, and probably only interesting for few.
In general all the above advice flo linked to includes most, but not all required to actually accomplish restoration of the functional experience prior to the loss. Perhaps the advice should be qualified to apply only when the supplemental packages to do with trust and dns are not in use. You might consider advising folks to use ipa-healthcheck for the purpose of deciding 'how bad the current situation is' on this or that system. Too is missing the advice relating to version management among 'masters' and 'replicas' (the order in which upgrades get installed among systems) and whether backups made by 'the last one' remain usable by 'the next one'.
ipa-healthcheck is not comprehensive and I don't want to leave the impression that if it passes then your install is a-ok. It consists of checks for common problems seen in deployments over the years but it doesn't check every possible thing that can go wrong.
I do believe that it is documented that a restore can only be done on the same version as the backup.
We do recommend keeping versions in sync.
The following is very valuable feedback on the documentation. The best way to get our doc writers attention on this is to use the feedback link. This will create internal tasks for them to tweak things that doesn't involve an e-mail that can be missed or forgotten from the devs.
Notice that the detail in the first link provided there makes the blanket statement affirming it is possible to replace a lost server by creating a new replica. However, did the author take responsibility for all of freeipa's parts when that was written? For example if the server holding the file kasp.db fails, does that statement yet apply? Is there a comprehensive list of 'files that are exceptions to the rule' maintained by anyone (sub account number ranges on replicas)?
Most IPA data is stored in LDAP therefore a new replica should be able to recover anything missing. The CA and KRA is the exception.
The next paragraph mentions VM snapshots, but leaves it to the reader to guess when 'a too outdated database' (4.2 sec 6) is or isn't the case. Too, does not mention that's not an answer if replication is active (timing issues, the documentation is missing about whether and when to walk users through the 're-init' Rob mentioned was required upstream). Might freeipa's replication setup itself notice when 'systemctl stop sssd;sss_cache -e;systemctl start sssd' is necessary and do that itself?
I believe the "too old" is restoring outside of the replication window which will freak it out. Assuming I'm understanding it right, this is what I meant by re-initing the world.
IPA should be independent from the SSSD cache.
The last paragraph mentions idM backups, detailed in section 5, which too I think might require a list of important files not backed up, and might too include Rob's advise that such is only useful when 'everything has burned to the ground and a fresh install on all systems is the next step'.
Maybe in an appendix. It is an absolutely huge list. Much of what is backed up consists of whole directories.
Section 3.2 might make more clear freeipa does not just have two states, 'master' and the term rhel uses 'regular replica' relevant to integrity, but at least six major variants relevant to integrity: replica without CA and not dnssec master, replica with ca but not dnssec master, etc. .. and 'master with ca and dnssec db', etc. It's likely this list may double in length in the case of KRA usage. And potentially need to be entirely revised in light of sub-domain account number range allocation. Also the freeipa instance mentioned in SOA records has a status of its own that needs thought in the 'promoting replica to master' advice.
What you're referring to is roles. It probably would be nice to add that the backup server should consist of all roles used by the system. If it were also the renewal master, crl generator, etc. then even better.
Does the advice of 'hidden replica' saving the day cover such files on the same list as kasp.db? If it's a manual step required as presently documented then ....?
As mentioned, kasp.db is backed up and I believe that ods-exporter will recover the keys from Custodia.
Last: As freeipa is itself in the 'policy' services sector, is not what ansible provides something the internal maps freeipa maintains a better place to capture and deploy that knowledge?
Perhaps. The backup/restore pre-dates Ansible. It would be an absolutely gigantic yaml file though. I don't know that Ansible would do things any better than the current python.
rob
On 10/19/23 10:45, Rob Crittenden wrote:
Harry G Coin via FreeIPA-users wrote:
On 10/18/23 07:30, Florence Blanc-Renaud via FreeIPA-users wrote:
Hi,
this guide explains the possible strategies for disaster recovery: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm...
And that one how to recover: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm...
Hope this helps, flo
Thanks Flo!
Fundamentally freeipa is having a tension in that restoration from system loss is almost never 'a committee or team effort' but a one person task--- but freeipa now has such richness that few people have the bandwidth to know all the necessary details for each of the distinct complex subdomains freeipa integrates. A goal yet to be attained is a freeipa 'best practice' advice that includes all the details necessary to restore prior functionality ( snapshot the vm? ipa-backup? the CA thing? Kasp.db? Subdomain account number ranges on replicas? Does promoting a replica fix the SOA record? Is when 'a database that's too old' a testable fact or like when your coffee temp is just right? ) . Freeipa ought to have within itself such maps as necessary to itself comprehend the intended role of the system being restored then maneuver and marshal such file changes 'from what it used to be' to 'what it needs to be' to make the system fill its intended role -- without requiring the system-restore person to be expert in each of freeipa's 'deep dive expertise file areas'.
This was quite the dense set of questions :-). I'll try to cover everything.
As far as I'm aware, ipa-backup and ipa-restore save and restore all things necessary to recover an IPA installation. This does hedge a bit. If, for example, you back up a server that doesn't have optional services installed, you could be in trouble. The CA, for example. The user has some responsibility in identifying the most critical server(s) to back up.
The DNSSEC keys are stored in LDAP so can be recovered, if missing, AFAIK. DNS isn't my area. But we do back up the kasp.db in any case, if DNS is configured.
I'm not sure what you mean regarding SOA. It is...ahem...problematic in that all servers may have different ones, but this is due to not having an atomic way to increment values over replication.
The rest of this post is detail on the above, and probably only interesting for few.
In general all the above advice flo linked to includes most, but not all required to actually accomplish restoration of the functional experience prior to the loss. Perhaps the advice should be qualified to apply only when the supplemental packages to do with trust and dns are not in use. You might consider advising folks to use ipa-healthcheck for the purpose of deciding 'how bad the current situation is' on this or that system. Too is missing the advice relating to version management among 'masters' and 'replicas' (the order in which upgrades get installed among systems) and whether backups made by 'the last one' remain usable by 'the next one'.
ipa-healthcheck is not comprehensive and I don't want to leave the impression that if it passes then your install is a-ok. It consists of checks for common problems seen in deployments over the years but it doesn't check every possible thing that can go wrong.
I do believe that it is documented that a restore can only be done on the same version as the backup.
We do recommend keeping versions in sync.
The following is very valuable feedback on the documentation. The best way to get our doc writers attention on this is to use the feedback link. This will create internal tasks for them to tweak things that doesn't involve an e-mail that can be missed or forgotten from the devs.
Notice that the detail in the first link provided there makes the blanket statement affirming it is possible to replace a lost server by creating a new replica. However, did the author take responsibility for all of freeipa's parts when that was written? For example if the server holding the file kasp.db fails, does that statement yet apply? Is there a comprehensive list of 'files that are exceptions to the rule' maintained by anyone (sub account number ranges on replicas)?
Most IPA data is stored in LDAP therefore a new replica should be able to recover anything missing. The CA and KRA is the exception.
The next paragraph mentions VM snapshots, but leaves it to the reader to guess when 'a too outdated database' (4.2 sec 6) is or isn't the case. Too, does not mention that's not an answer if replication is active (timing issues, the documentation is missing about whether and when to walk users through the 're-init' Rob mentioned was required upstream). Might freeipa's replication setup itself notice when 'systemctl stop sssd;sss_cache -e;systemctl start sssd' is necessary and do that itself?
I believe the "too old" is restoring outside of the replication window which will freak it out. Assuming I'm understanding it right, this is what I meant by re-initing the world.
IPA should be independent from the SSSD cache.
How may non-developers know what 'the replication window' is in minutes for their setup? Does any part of the restore process notice this and tell the operator, it's a lot to expect someone given the restore job to 'just know' what the 'replication window' might be? Also, given the documentation says when this occurs steps on all replicas and masters related to sssd cache must be taken, why then doesn't freeipa notice the current time is outside the replication window for the current system to 're-init itself', why doesn't all the affected systems notice and take the sssd steps as documented as each replica gets information via replication traffic?
The last paragraph mentions idM backups, detailed in section 5, which too I think might require a list of important files not backed up, and might too include Rob's advise that such is only useful when 'everything has burned to the ground and a fresh install on all systems is the next step'.
Maybe in an appendix. It is an absolutely huge list. Much of what is backed up consists of whole directories.
Section 3.2 might make more clear freeipa does not just have two states, 'master' and the term rhel uses 'regular replica' relevant to integrity, but at least six major variants relevant to integrity: replica without CA and not dnssec master, replica with ca but not dnssec master, etc. .. and 'master with ca and dnssec db', etc. It's likely this list may double in length in the case of KRA usage. And potentially need to be entirely revised in light of sub-domain account number range allocation. Also the freeipa instance mentioned in SOA records has a status of its own that needs thought in the 'promoting replica to master' advice.
What you're referring to is roles. It probably would be nice to add that the backup server should consist of all roles used by the system. If it were also the renewal master, crl generator, etc. then even better.
Does the advice of 'hidden replica' saving the day cover such files on the same list as kasp.db? If it's a manual step required as presently documented then ....?
As mentioned, kasp.db is backed up and I believe that ods-exporter will recover the keys from Custodia.
Notice https://www.freeipa.org/page/Howto/DNSSEC --- waaaay at the bottom, reveals there are "freeipa regular masters" and "freeipa dnssec masters" -- either the official documentation is giving bad advice as current, or t's not enough to do as the documentation flatly states is possible and just promote a replica.
It's this sort of thing. It's about whether operating freeipa requires a person with development level expertise in every subsystem to be a sustainable option in an organization, whether the documentation about critical things like restoring previous functionality is complete and correct and can be trusted, or whether even after 'doing what the docs says in the correct fashion' nevertheless one finds the system is not actually doing what it did beforehand.
You might consider storing a shared file that's accessible with zero assumptions about the current state of any freeipa related packages or configurations, human readable, with a timestamp in it. Listing by fqdn the 'actual for real intended role' of only the hosts running freeipa in some fashion, to indicate this 'special status realities' like 'dnssec master' 'ca authority' 'account subranges' 'replica' etc. etc. Then 'install time' and 'restore' time from the user's perspective is the same: ask the user whether the timestamp in the 'freeipa enterprise map file' is correct and the latest -- then freeipa will take it upon itself to install, locate, update, fetch, crunch whatever it has with whatever it needs to turn the current host into what the map says it's supposed to do.
On 10/17/23 12:50, Christian Heimes via FreeIPA-users wrote:
On 17/10/2023 19.32, Harry G Coin via FreeIPA-users wrote:
'security' and 'other' seemingly 'unrelated' 'upgrades' to packages n levels deep but whose previously un-noticed freeipa killing race-condition or other bug manifests after the upgrade. I find myself obligated to prevent any security or other change from happening until the lowest possible usage times. For example today's 'random freeipa bother' is:
Problem: cannot install both protobuf-3.5.0-15.el8.x86_64 and protobuf-3.19.0-2.el8s.x86_64 - package liborc1-1.7.9-1.el8.x86_64 requires libprotobuf.so.15()(64bit), but none of the providers can be installed - cannot install the best update candidate for package protobuf-3.19.0-2.el8s.x86_64 - cannot install the best update candidate for package liborc1-1.7.5-1.el8s.x86_64 (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
How did you end up with Hadoop-related libraries on your IPA server? Did you install additional services and EPEL on your IPA server?
The enormous project ceph, (details at https://ceph.io ) offers high availability files systems across racks of servers as well as multiple drives per server, offers a file system 'cephfs'. To get access to shared directories by the native cephfs in the rhel world one must install the ceph-common package. Seems freeipa doesn't play well with those variants.
Christian
In order to connect to a ceph fs share, I need to install ceph_common. Seems
-- Christian Heimes Principal Software Engineer, Identity Management and Platform Security
Red Hat GmbH,https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
FreeIPA-users mailing list --freeipa-users@lists.fedorahosted.org To unsubscribe send an email tofreeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
freeipa-users@lists.fedorahosted.org
-
Christian Heimes -
Florence Blanc-Renaud -
Harry G Coin -
Rob Crittenden