FreeIPA cluster, backup and restore

[Finn Fysj]

Have a cluster setup with is setup using Ansible FreeIPA roles ipaserver &
ipareplica.
Running ipabackup using script as the ipabackup role doesn't work as wanted or
intended, meaning not able to take backup of data. 

Multiple master, only one with CA installed. 
When I run ipabackup to backup data I get the following:

Error: Local roles  do not match globally used roles CA. A backup done on this host would
not be complete enough to restore a fully functional, identical cluster.
The ipa-backup command failed. See /var/log/ipabackup.log for more information.

The error message is somewhat understandable. We don't use FreeIPA CA capabilities, so
that's the reason we don't have it installed on replicas, unless you guys would
recommend otherwise?


I've tried to test a little using these ansible roles. What happens if my Master with
the only backup goes down? Yes, I'll have a replica making sure everything works as
normal, so I can scrap the master, rebuild it and restore the data backup I took.
However, once the node is restored, there's still not any connection between the two
nodes now, since a re-run of the ipareplica won't do anything since it's already
installed. Does that mean we need to rebuild this node as well?

A normal data restore of a node will stop the replication connection between the two
nodes, meaning it needs to be "re-connected", this is also not something that
can be done using these roles?

One final question: If we have a working cluster setup, and some sausage fingers manages
to delete the replica from the "CA node". How can I re-initalize this with the
ansible replica role, or is rebuild the only option?

[Yavor Marinov]

Hey Finn,

for our replications where we don't have any CA installed i'm using the
following ipabackup options to have proper backup:

ipa-backup --disable-role-check --logs --quiet

As I had to prepare a disaster recovery plan, I've tested both masters and
replicas in a lab in order to evaluate the backup procedures.

On Tue, Jan 30, 2024 at 1:56 PM Finn Fysj via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org&gt; wrote:

>  Have a cluster setup with is setup using Ansible FreeIPA roles
> ipaserver &
>  ipareplica.
>  Running ipabackup using script as the ipabackup role doesn't work as
>  wanted or intended, meaning not able to take backup of data.
>
>  Multiple master, only one with CA installed.
>  When I run ipabackup to backup data I get the following:
>
>  Error: Local roles  do not match globally used roles CA. A backup done on
>  this host would not be complete enough to restore a fully functional,
>  identical cluster.
>  The ipa-backup command failed. See /var/log/ipabackup.log for more
>  information.
>
>  The error message is somewhat understandable. We don't use FreeIPA CA
>  capabilities, so that's the reason we don't have it installed on replicas,
>  unless you guys would recommend otherwise?
>
>
>  I've tried to test a little using these ansible roles. What happens if my
>  Master with the only backup goes down? Yes, I'll have a replica making sure
>  everything works as normal, so I can scrap the master, rebuild it and
>  restore the data backup I took.
>  However, once the node is restored, there's still not any connection
>  between the two nodes now, since a re-run of the ipareplica won't do
>  anything since it's already installed. Does that mean we need to rebuild
>  this node as well?
>
>  A normal data restore of a node will stop the replication connection
>  between the two nodes, meaning it needs to be "re-connected", this is also
>  not something that can be done using these roles?
>
>  One final question: If we have a working cluster setup, and some sausage
>  fingers manages to delete the replica from the "CA node". How can I
>  re-initalize this with the ansible replica role, or is rebuild the only
>  option?
>  --
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>

[Finn Fysj]

>  Hey Finn,
>  
>  for our replications where we don't have any CA installed i'm using the
>  following ipabackup options to have proper backup:
>  
>  ipa-backup --disable-role-check --logs --quiet 
Cheers Yavor. I'll have a look
at this. 
However it only answers a part of my problem. You'll face a issue with the RUV error
with the replica having a different database generation ID. Meaning it needs to be
re-initialized, right?

I see the ipatopologysuffix module doesn't have a way to check for working replication
between the nodes.

[Rob Crittenden]

Finn Fysj via FreeIPA-users wrote:
> > Hey Finn,
> >
> > for our replications where we don't have any CA installed i'm using the
> > following ipabackup options to have proper backup:
> >
> > ipa-backup --disable-role-check --logs --quiet
>  Cheers Yavor. I'll have a look at this. 
>  However it only answers a part of my problem. You'll face a issue with the RUV error
> with the replica having a different database generation ID. Meaning it needs to be
> re-initialized, right?
>  
>  I see the ipatopologysuffix module doesn't have a way to check for working
> replication between the nodes. 
I think there is some misunderstanding about the purpose of backup and
restore. It is for catastrophic recovery only.

This is why it wants all roles to be included because if you lose your
cluster and the only backup you have is lacking a role, say the CA, that
is less than awesome.

A restore will disable all replication agreements. They can be
re-enabled but a restore by its very nature is going back in time which
is going to confuse the heck out of replication. At best any other
existing servers will need to be re-initialized. Otherwise they need to
be re-installed. Remember: catastrophic.

It is not designed for recovering a single entry that was accidentally
deleted, or an undesired edit. If periodic backups are done the data is
available in the stored LDIFs but it is an exercise for the user to
restore in that case.

rob

[Finn Fysj]

>  Finn Fysj via FreeIPA-users wrote:
>  
>  I think there is some misunderstanding about the purpose of backup and
>  restore. It is for catastrophic recovery only.
>  
>  This is why it wants all roles to be included because if you lose your
>  cluster and the only backup you have is lacking a role, say the CA, that
>  is less than awesome.
>  
>  A restore will disable all replication agreements. They can be
>  re-enabled but a restore by its very nature is going back in time which
>  is going to confuse the heck out of replication. At best any other
>  existing servers will need to be re-initialized. Otherwise they need to
>  be re-installed. Remember: catastrophic.
>  
>  It is not designed for recovering a single entry that was accidentally
>  deleted, or an undesired edit. If periodic backups are done the data is
>  available in the stored LDIFs but it is an exercise for the user to
>  restore in that case.
>  
>  rob 
I appreciate so much for your response. 

I've experienced issues where I tried to uninstall the replica server and trying to
re-installing:

  roles:
    - role: freeipa.ansible_freeipa.ipareplica
      ipaserver_ignore_topology_disconnect: true
      ipaserver_ignore_last_of_role: true
      state: absent

Then I try to install it again:
    - role: freeipa.ansible_freeipa.ipareplica
      ipaclient_force_join: true
      ipareplica_install_packages: true
      ipareplica_setup_firewalld: false
      ipareplica_setup_dns: false
      ipareplica_servers: master1.example.com
      ipareplicas: ["{{ ansible_play_hosts_all | join(', ') }}"]
      ipareplica_domain: "example.com"
      ipaadmin_principal: "admin"
      ipaadmin_password: "Secret1213"
      ipadm_password: "my_password"


I run into issues such as:

"module_stdout": "Traceback (most recent call last):\r\n  File
\"/usr/lib/python3.9/site-packages/ipalib/krb_utils.py\", line 182, in
get_principal\r\n    creds = get_credentials(ccache_name=ccache_name)\r\n  File
\"/usr/lib/python3.9/site-packages/ipalib/krb_utils.py\", line 165, in
get_credentials\r\n    return gssapi.Credentials(usage=\"initiate\", name=name,
store=store)\r\n  File \"/usr/lib64/python3.9/site-packages/gssapi/creds.py\",
line 63, in __new__\r\n    res = cls.acquire(name, lifetime, mechs, usage,\r\n  File
\"/usr/lib64/python3.9/site-packages/gssapi/creds.py\", line 136, in acquire\r\n
   res = rcreds.acquire_cred(name, lifetime,\r\n  File \"gssapi/raw/creds.pyx\",
line 161, in
gssapi.raw.creds.acquire_cred\r\ngssapi.raw.exceptions.MissingCredentialsError: Major
(458752): No credentials were supplied, or the credentials were unavailable or
inaccessible, Minor (2529639053): No Kerberos credentials available (default cache:
)\r\n\r\nDuring handling of the above excepti
 on, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File
\"/home/ansible/.ansible/tmp/ansible-tmp-1706882379.2962139-14800-182642519268134/AnsiballZ_ipareplica_add_to_ipaservers.py\",
line 107, in <module>\r\n    _ansiballz_main()\r\n  File
\"/home/ansible/.ansible/tmp/ansible-tmp-1706882379.2962139-14800-182642519268134/AnsiballZ_ipareplica_add_to_ipaservers.py\",
line 99, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path,
ANSIBALLZ_PARAMS)\r\n  File
\"/home/ansible/.ansible/tmp/ansible-tmp-1706882379.2962139-14800-182642519268134/AnsiballZ_ipareplica_add_to_ipaservers.py\",
line 47, in invoke_module\r\n   
runpy.run_module(mod_name='ansible_collections.freeipa.ansible_freeipa.plugins.modules.ipareplica_add_to_ipaservers',
init_globals=dict(_module_fqn='ansible_collections.freeipa.ansible_freeipa.plugins.modules.ipareplica_add_to_ipaservers',
_modlib_path=modlib_path),\r\n  File \"/usr/lib64/python3.9/runpy.py\", line
225, in run_module\r\n   
  return _run_module_code(code, init_globals, run_name, mod_spec)\r\n  File
\"/usr/lib64/python3.9/runpy.py\", line 97, in _run_module_code\r\n   
_run_code(code, mod_globals, init_globals,\r\n  File
\"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code\r\n    exec(code,
run_globals)\r\n  File
\"/tmp/ansible_freeipa.ansible_freeipa.ipareplica_add_to_ipaservers_payload_gnneon23/ansible_freeipa.ansible_freeipa.ipareplica_add_to_ipaservers_payload.zip/ansible_collections/freeipa/ansible_freeipa/plugins/modules/ipareplica_add_to_ipaservers.py\",
line 156, in <module>\r\n  File
\"/tmp/ansible_freeipa.ansible_freeipa.ipareplica_add_to_ipaservers_payload_gnneon23/ansible_freeipa.ansible_freeipa.ipareplica_add_to_ipaservers_payload.zip/ansible_collections/freeipa/ansible_freeipa/plugins/modules/ipareplica_add_to_ipaservers.py\",
line 139, in main\r\n  File
\"/usr/lib/python3.9/site-packages/ipalib/backend.py\", line 69, in connect\r\n 
  conn = self.create_connection(*args, **kw)\r\n  Fil
 e \"/usr/lib/python3.9/site-packages/ipaserver/plugins/ldap2.py\", line 203, in
create_connection\r\n    principal = krb_utils.get_principal(ccache_name=ccache)\r\n  File
\"/usr/lib/python3.9/site-packages/ipalib/krb_utils.py\", line 185, in
get_principal\r\n    raise
errors.CCacheError(message=str(e))\r\nipalib.errors.CCacheError: Major (458752): No
credentials were supplied, or the credentials were unavailable or inaccessible, Minor
(2529639053): No Kerberos credentials available (default cache: )\r\n",

Hello,

On 1/30/24 12:56, Finn Fysj via FreeIPA-users wrote:
>  Have a cluster setup with is setup using Ansible FreeIPA roles
> ipaserver & ipareplica.
>  Running ipabackup using script as the ipabackup role doesn't work as wanted or
> intended, meaning not able to take backup of data.
>  what kind of data is missing in the backup using the role? 
With the same options, the ipabackup role should do exactly the same as the 
command line tool. The role is using the command line tool internally.

>  Multiple master, only one with CA installed.
>  When I run ipabackup to backup data I get the following:
>  
>  Error: Local roles  do not match globally used roles CA. A backup done on this host would
> not be complete enough to restore a fully functional, identical cluster.
>  The ipa-backup command failed. See /var/log/ipabackup.log for more information.
>  
>  The error message is somewhat understandable. We don't use FreeIPA CA capabilities,
> so that's the reason we don't have it installed on replicas, unless you guys would
> recommend otherwise?
>  
>  
>  I've tried to test a little using these ansible roles. What happens if my Master with
> the only backup goes down? Yes, I'll have a replica making sure everything works as
> normal, so I can scrap the master, rebuild it and restore the data backup I took.
>  However, once the node is restored, there's still not any connection between the two
> nodes now, since a re-run of the ipareplica won't do anything since it's already
> installed. Does that mean we need to rebuild this node as well?
>  
>  A normal data restore of a node will stop the replication connection between the two
> nodes, meaning it needs to be "re-connected", this is also not something that
> can be done using these roles?
>  
>  One final question: If we have a working cluster setup, and some sausage fingers manages
> to delete the replica from the "CA node". How can I re-initalize this with the
> ansible replica role, or is rebuild the only option?
>  --
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Finn Fysj]

>  Hello,
>  
>  On 1/30/24 12:56, Finn Fysj via FreeIPA-users wrote:
>  
>  With the same options, the ipabackup role should do exactly the same as the 
>  command line tool. The role is using the command line tool internally. 
Yes... Would
be nice if the ansible role would've been a little more verbose when doing a data
restore, to include something like: 
  The replication agreements on\nmasters running IPA 3.1 or earlier will need to be
manually\nre-enabled. 
  See the man page for details.\nDisabling all replication.\nDisabling replication
agreement on...



Also is there a way to make ipa skip authselect configuration when restoring backup, like
upgrade?
[authcfg]
migrated_to_authselect = True