Design discussion - Changes required to support one-way trusts

[Jakub Hrozek]

Hi,

I prepared design page for the one-way trusts feature:
    https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts

The text is copied below as well for your convenience. I have a couple
of questions to discuss:
    - Is it worth it to put the keytabs into a new subdirectory so that
      a special SELinux label can be applied? I think yes, since then
      we'd have additional layer of protection of the trust secrets
    - Is there any reasonable fallback beyond re-fetching the keytab we
      can do? I can't think of any..
    - Should the IPA tools signal SSSD either via UNIX signals or via
      the DBus API once a trust has been added or removed to minimize
      the race condition and improve user experience? Currently I hope
      not, the proposed mechanism should be enough but testing is still
      required.

When the design page is agreed on, I'll turn the three subsections of
the implementation phase into smaller tickets and close the tracker.

= Feature Name =

Related ticket(s):
 * https://fedorahosted.org/sssd/ticket/2579

=== Problem statement ===
The next IPA release will support one-way trusts. SSSD needs to add support
for this feature in its server mode.

=== Use cases ===
One-way trust to Active Directory where FreeIPA realm trusts Active Directory
forest using cross-forest trust feature of AD but the AD forest does not trust
FreeIPA realm. Users from AD forest can access resources in FreeIPA realm.

=== Overview of the solution ===
At a high level, SSSD needs to examine the trust objects whether they
are one-way or two way trusts. For each one-way trust, SSSD needs to fetch
and store the keytab and use the keytab to secure the connection. For
two-way trusts, we can keep using the existing code that reuses the IPA
realm and the system keytab for both IPA and AD connectins. Care must be
taken to remove keytabs of trusts that were removed as well.

Fetching the keytab would be done by calling the `ipa-getkeytab` utility
for every one-way trust. The keytab would only be (re)fetched if it's
missing or if attempting to use the keytab failed. On the IPA server,
we must make sure that the IPA server identity is allowed to read the keytab.

Because handling multiple keytabs increases the risk of failing connections in
case the trust wasn't setup correctly, we need to modify the failover code to
not set the whole back end offline in case connecting to a subdomain AD server
fails. Instead, the subdomain will be marked as inactive for a short period of
time, during which it would act as offline. The proper way of solving this
problem would be to rework the failover module so that it can act per domain,
not only per back end, however, that change is out of scope for this release.

=== Implementation details ===
This section describes all the required changes in detail.

==== Reading the subdomains in the IPA subdomain handler ====
The IPA subdomain handler will include the attribute `ipaNTTrustDirection`
when reading the trust objects. Currently this attribute is not readable
by the host principal, so the IPA ACIs must be relaxed (ipa ticket?). If
the trust direction is set to an OR of `lsa.LSA_TRUST_DIRECTION_OUTBOUND`
and `lsa.LSA_TRUST_DIRECTION_INBOUND`, then it's a two-way trust and we'll
just use the existing code that re-uses the IPA keytab for the AD trusted
domain as well. If the attribute is only `lsa.LSA_TRUST_DIRECTION_OUTBOUND`,
we handle the trust as a one-way trust. The trust type can be stored in
`ipa_ad_server_ctx`.

Each `sss_domain_info` structure will be created as `inactive` in the
subdomain code. After enumerating the trusted domains, the subdomain
handler will check if a keytab already exists for every one-way trusted
domain. If yes, the domain is ready to use and can be enabled. If there is
no keytab, the subdomain handler will fork out a call to `ipa-getkeytab`,
fetch the keytab and store it under `/var/lib/sss/db`. The `ipa-getkeytab`
call will be done using Kerberos credentials the host has. IPA ACIs must
be modified accordingly to allow the IPA server principals to fetch the
trust keytabs, but noone else.

The directory `/var/lib/sss/db` is already only accessible to the sssd
user only. As an additional security measure, we might want to store the
keytabs under a new directory so that the keytabs also receive a SELinux
context stricter than the default `sssd_var_lib_t`. If fetching the keytab
succeeds, the domain would be enabled.  The SELinux policy must also be
adjusted to allow calling `ipa-getkeytab` by the `sssd_be` process.

If any trust relationships were removed, the corresponding keytabs must
be removed from the disk as well.

==== Changes to the AD id_ctx instantiation ====
With two-way trust, we can keep using the default IPA principal and keytab.

With one-way trust, the keytab retrieved from the IPA server must be
used. Also, the principal must be passed into the `ad_create_default_options`
function. The custom values must be set before we proceed to instantiate
LDAP provider options. The only AD provider option we need to set is
AD_KRB5_REALM.

In the LDAP provider, we must take care that the following sdap options
are set correctly:
    * SDAP_SASL_AUTHID - must be set to the NetBIOS name of the IPA domain. (A
      domain `TRUST.COM` would set this value to `TRUST$`).
    * SDAP_SASL_REALM - must be set to the AD realm
    * SDAP_KRB5_KEYTAB - must be set to the per-domain keytab retrieved from IPA

The AD provider eventually calls `sdap_set_sasl_options()` from the
LDAP provider, we need to make sure this function receives the correct
values. During experimentation we were able to show that using multiple
different SASL users and different realms doesn't cause any problems in
SASL or LDAP libraries.

The only place that will keep using the IPA realm is the failover
instantiation. We need to keep using this hack until failover is per-backend.

==== Subdomain offline status changes ====
At the moment, the whole back end can be either online or offline and
the status applies to both the main domain and the subdomains alike. As
an effect, a failure to connect to a subdomain server would also make the
main domain operate offline. In many subdomain setups, it's actually more
convenient not to, because the subdomain server might be on a different
network segment, behind a different firewall etc. Instead, the domain
should only be made inactive.

The `sss_domain_info` structure would convert the 'bool disabled' parameter
into an `enum sss_domain_state`. The supported values would be:
    * ''disabled'' - the domain should not be used by either responder or
      provider. It was removed or disabled on the server.
    * ''active'' - the domain can be used by a responder and the data
provider
      would forward its request to the backend
    * ''inactive'' - the domain can be used by a responder, but the data
      provider would just shortcut as if the domain was offline. For now, this
      option will be used by subdomains only.

The implementation would include renaming the existing `be_mark_offline()`
function to be called `be_mark_dom_offline()` and modifying its behaviour. The
existing code that sets the offline status and runs the offline callbacks
would be called for parent domains only. For subdomains, we would mark the
subdomain as inactive and schedule a tevent request that would unconditionally
reset inactive domain to active. The request would be scheduled after
`offline_timeout` seconds to be consistent with main domains from user's
perspective. Likewise, the `be_reset_offline()` function will be extended to
reset inactive domains to active as well as the SIGUSR1 and SIGUSR2 signal
handlers. Finally, all calls to the `be_is_offline()` function should be
inspected and the invocations that are per-domain should be converted to a new
function `be_dom_is_offline()` that would check the offline status for parent
domains and the offline state for subdomains. We should also make sure the
backend offline status structure is opaque as currently its internals are
readable by external users as well. Making the offline status opaque would
make it safer to perform modifications to the offline code.

In both offline and inactive cases, the ID handlers would reply with
`DP_ERR_OFFLINE`. The crucial difference between offline and inactive at this
point would be that inactive domains are re-activated undonditionally. When we
modify the failover code to handle domains separately, we'll be able to
leverage per-domain online checks or online/offline callbacks as well.

==== Detecting re-established trusts ====
In cases the `sdap_kinit_send()` request fails, the sdap code would return a
special error code instead of blindly returning `EIO` as it does at the
moment. When the `ipa_get_ad_acct` request receives this error code, it would
re-run the subdomain request in order to check if the trust relationship still
exists and in order to re-fetch the keytab again. In case the trust is still
established, we would re-set the inactive domain status and re-run the account
request. In case the trust went away, the subdomain code should remove the
trusted domain (however, this must be tested). If the account request returns
any error after reinitializing the ketab, the subdomain would be marked
inactive in order to avoid fetching the keytab again and again. In order to be
able to run the subdomain request separately from the subdomain back end
handler, the subdomain code must be wrapped into a subdomain request as the
code currently assumes it's being called from the subdomain backend handler
only.

=== Future work ===
Handling failover and offline status on per-domain basis instead of
per-backend basis should be done in the next release.

=== Configuration changes ===
none

=== How To Test ===
Establish a one-way trust relationship with an AD domain. Make sure both
IPA and AD users are resolvable. It's prudent to test combinations of
one-way and two-way trusts with different forests. Make sure removing a
trust relationship removes the keytab from the filesystem. Make sure that
SSSD handles re-establishing a trust relationship.

=== Authors ===
* Jakub Hrozek <jhrozek(a)redhat.com&gt;

[Simo Sorce]

Very nice writeup!
A few comments..

On Sun, 2015-04-26 at 21:22 +0200, Jakub Hrozek wrote:
>  Hi,
>  
>  I prepared design page for the one-way trusts feature:
>      https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts
>  
>  The text is copied below as well for your convenience. I have a couple
>  of questions to discuss:
>      - Is it worth it to put the keytabs into a new subdirectory so that
>        a special SELinux label can be applied? I think yes, since then
>        we'd have additional layer of protection of the trust secrets 
It is not clear to me what would be the advantage, who are you trying to
protect against ?
Adding different labels makes sense only if sssd_be has a different
label from the other sssd processes, but then you can simply have
selinux policies that label accordingly to the parent process in the
same directory, no?

>      - Is there any reasonable fallback beyond re-fetching the keytab
> we
>        can do? I can't think of any.. 
Be careful with refetches and so we do not want to end up in infinite
loops, where the keytab can be fetched but still operation fails for
whatever reason.
One thing that should be checked is if the new keytab and old keytab
have the same keys in them, and, if so just stop re-fetching and raising
a fatal error (at list for a reasonable period of time).

>      - Should the IPA tools signal SSSD either via UNIX signals or
> via
>        the DBus API once a trust has been added or removed to minimize
>        the race condition and improve user experience? Currently I hope
>        not, the proposed mechanism should be enough but testing is still
>        required.
>  
>  When the design page is agreed on, I'll turn the three subsections of
>  the implementation phase into smaller tickets and close the tracker.
>  
>  = Feature Name =
>  
>  Related ticket(s):
>   * https://fedorahosted.org/sssd/ticket/2579
>  
>  === Problem statement ===
>  The next IPA release will support one-way trusts. SSSD needs to add support
>  for this feature in its server mode.
>  
>  === Use cases ===
>  One-way trust to Active Directory where FreeIPA realm trusts Active Directory
>  forest using cross-forest trust feature of AD but the AD forest does not trust
>  FreeIPA realm. Users from AD forest can access resources in FreeIPA realm.
>  
>  === Overview of the solution ===
>  At a high level, SSSD needs to examine the trust objects whether they
>  are one-way or two way trusts. For each one-way trust, SSSD needs to fetch
>  and store the keytab and use the keytab to secure the connection. For
>  two-way trusts, we can keep using the existing code that reuses the IPA
>  realm and the system keytab for both IPA and AD connectins. Care must be
>  taken to remove keytabs of trusts that were removed as well.
>  
>  Fetching the keytab would be done by calling the `ipa-getkeytab` utility
>  for every one-way trust. The keytab would only be (re)fetched if it's
>  missing or if attempting to use the keytab failed. On the IPA server,
>  we must make sure that the IPA server identity is allowed to read the keytab.
>  
>  Because handling multiple keytabs increases the risk of failing connections in
>  case the trust wasn't setup correctly, we need to modify the failover code to
>  not set the whole back end offline in case connecting to a subdomain AD server
>  fails. Instead, the subdomain will be marked as inactive for a short period of
>  time, during which it would act as offline. The proper way of solving this
>  problem would be to rework the failover module so that it can act per domain,
>  not only per back end, however, that change is out of scope for this release.
>  
>  === Implementation details ===
>  This section describes all the required changes in detail.
>  
>  ==== Reading the subdomains in the IPA subdomain handler ====
>  The IPA subdomain handler will include the attribute `ipaNTTrustDirection`
>  when reading the trust objects. Currently this attribute is not readable
>  by the host principal, so the IPA ACIs must be relaxed (ipa ticket?). If
>  the trust direction is set to an OR of `lsa.LSA_TRUST_DIRECTION_OUTBOUND`
>  and `lsa.LSA_TRUST_DIRECTION_INBOUND`, then it's a two-way trust and we'll
>  just use the existing code that re-uses the IPA keytab for the AD trusted
>  domain as well. If the attribute is only `lsa.LSA_TRUST_DIRECTION_OUTBOUND`,
>  we handle the trust as a one-way trust. The trust type can be stored in
>  `ipa_ad_server_ctx`. 
What do you do if you have only lsa.LSA_TRUST_DIRECTION_INBOUND ?

>  Each `sss_domain_info` structure will be created as `inactive` in
> the
>  subdomain code. After enumerating the trusted domains, the subdomain
>  handler will check if a keytab already exists for every one-way trusted
>  domain. If yes, the domain is ready to use and can be enabled. If there is
>  no keytab, the subdomain handler will fork out a call to `ipa-getkeytab`,
>  fetch the keytab and store it under `/var/lib/sss/db`. The `ipa-getkeytab`
>  call will be done using Kerberos credentials the host has. IPA ACIs must
>  be modified accordingly to allow the IPA server principals to fetch the
>  trust keytabs, but noone else.
>  
>  The directory `/var/lib/sss/db` is already only accessible to the sssd
>  user only. As an additional security measure, we might want to store the
>  keytabs under a new directory so that the keytabs also receive a SELinux
>  context stricter than the default `sssd_var_lib_t`. If fetching the keytab
>  succeeds, the domain would be enabled.  The SELinux policy must also be
>  adjusted to allow calling `ipa-getkeytab` by the `sssd_be` process. 
Getting a keytab is an LDAP extended operation, do you want to fork off
ipa-getkeytab to avoid coding the LDAP operatin in SSSD ? Or for some
other reason ?

>  If any trust relationships were removed, the corresponding keytabs
> must
>  be removed from the disk as well. 
How do you do this ?
Just regularly enumerate the keytab files on disk and remove the ones
that have no object in the cache ?
One advantage of using a separate subdir for keytabs, as you propose,
would be to wipe it clean every time sssd restart (at stop and at start
too for good measure), then enumerate the trusts on the server and
re-fetch any keytab.

>  ==== Changes to the AD id_ctx instantiation ====
>  With two-way trust, we can keep using the default IPA principal and keytab.
>  
>  With one-way trust, the keytab retrieved from the IPA server must be
>  used. Also, the principal must be passed into the `ad_create_default_options`
>  function. The custom values must be set before we proceed to instantiate
>  LDAP provider options. The only AD provider option we need to set is
>  AD_KRB5_REALM.
>  
>  In the LDAP provider, we must take care that the following sdap options
>  are set correctly:
>      * SDAP_SASL_AUTHID - must be set to the NetBIOS name of the IPA domain. (A
>        domain `TRUST.COM` would set this value to `TRUST$`). 
The NetBIOS domain can be completely unrelated to the DNS domain, so
make sure you do not just try to truncate the realm but actually fetch
the info from IPA/AD.

>      * SDAP_SASL_REALM - must be set to the AD realm
>      * SDAP_KRB5_KEYTAB - must be set to the per-domain keytab retrieved from IPA
>  
>  The AD provider eventually calls `sdap_set_sasl_options()` from the
>  LDAP provider, we need to make sure this function receives the correct
>  values. During experimentation we were able to show that using multiple
>  different SASL users and different realms doesn't cause any problems in
>  SASL or LDAP libraries.
>  
>  The only place that will keep using the IPA realm is the failover
>  instantiation. We need to keep using this hack until failover is per-backend.
>  
>  ==== Subdomain offline status changes ====
>  At the moment, the whole back end can be either online or offline and
>  the status applies to both the main domain and the subdomains alike. As
>  an effect, a failure to connect to a subdomain server would also make the
>  main domain operate offline. In many subdomain setups, it's actually more
>  convenient not to, because the subdomain server might be on a different
>  network segment, behind a different firewall etc. Instead, the domain
>  should only be made inactive. 
Does this cover accurately the Forest vs AD subdomain cases ?
We will have a keytab only for the AD root domain (the forest) and we
need to use those credentials for any other domain in the AD forest.

>  The `sss_domain_info` structure would convert the 'bool
> disabled' parameter
>  into an `enum sss_domain_state`. The supported values would be:
>      * ''disabled'' - the domain should not be used by either responder
> or
>        provider. It was removed or disabled on the server.
>      * ''active'' - the domain can be used by a responder and the data
> provider
>        would forward its request to the backend
>      * ''inactive'' - the domain can be used by a responder, but the data
>        provider would just shortcut as if the domain was offline. For now, this
>        option will be used by subdomains only.
>  
>  The implementation would include renaming the existing `be_mark_offline()`
>  function to be called `be_mark_dom_offline()` and modifying its behaviour. The
>  existing code that sets the offline status and runs the offline callbacks
>  would be called for parent domains only. For subdomains, we would mark the
>  subdomain as inactive and schedule a tevent request that would unconditionally
>  reset inactive domain to active. The request would be scheduled after
>  `offline_timeout` seconds to be consistent with main domains from user's
>  perspective. Likewise, the `be_reset_offline()` function will be extended to
>  reset inactive domains to active as well as the SIGUSR1 and SIGUSR2 signal
>  handlers. Finally, all calls to the `be_is_offline()` function should be
>  inspected and the invocations that are per-domain should be converted to a new
>  function `be_dom_is_offline()` that would check the offline status for parent
>  domains and the offline state for subdomains. We should also make sure the
>  backend offline status structure is opaque as currently its internals are
>  readable by external users as well. Making the offline status opaque would
>  make it safer to perform modifications to the offline code.
>  
>  In both offline and inactive cases, the ID handlers would reply with
>  `DP_ERR_OFFLINE`. The crucial difference between offline and inactive at this
>  point would be that inactive domains are re-activated undonditionally. When we
>  modify the failover code to handle domains separately, we'll be able to
>  leverage per-domain online checks or online/offline callbacks as well.
>  
>  ==== Detecting re-established trusts ====
>  In cases the `sdap_kinit_send()` request fails, the sdap code would return a
>  special error code instead of blindly returning `EIO` as it does at the
>  moment. When the `ipa_get_ad_acct` request receives this error code, it would
>  re-run the subdomain request in order to check if the trust relationship still
>  exists and in order to re-fetch the keytab again. In case the trust is still
>  established, we would re-set the inactive domain status and re-run the account
>  request. In case the trust went away, the subdomain code should remove the
>  trusted domain (however, this must be tested). If the account request returns
>  any error after reinitializing the ketab, the subdomain would be marked
>  inactive in order to avoid fetching the keytab again and again. In order to be
>  able to run the subdomain request separately from the subdomain back end
>  handler, the subdomain code must be wrapped into a subdomain request as the
>  code currently assumes it's being called from the subdomain backend handler
>  only.
>  
>  === Future work ===
>  Handling failover and offline status on per-domain basis instead of
>  per-backend basis should be done in the next release.
>  
>  === Configuration changes ===
>  none
>  
>  === How To Test ===
>  Establish a one-way trust relationship with an AD domain. Make sure both
>  IPA and AD users are resolvable. It's prudent to test combinations of
>  one-way and two-way trusts with different forests. Make sure removing a
>  trust relationship removes the keytab from the filesystem. Make sure that
>  SSSD handles re-establishing a trust relationship.
>  
>  === Authors ===
>  * Jakub Hrozek <jhrozek(a)redhat.com&gt; 
Good job.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Jakub Hrozek]

On Sun, Apr 26, 2015 at 06:17:21PM -0400, Simo Sorce wrote:
>  Very nice writeup!
>  A few comments..
>  
>  On Sun, 2015-04-26 at 21:22 +0200, Jakub Hrozek wrote:
>  > Hi,
>  > 
>  > I prepared design page for the one-way trusts feature:
>  >     https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts
>  > 
>  > The text is copied below as well for your convenience. I have a couple
>  > of questions to discuss:
>  >     - Is it worth it to put the keytabs into a new subdirectory so that
>  >       a special SELinux label can be applied? I think yes, since then
>  >       we'd have additional layer of protection of the trust secrets
>  
>  It is not clear to me what would be the advantage, who are you trying to
>  protect against ?
>  Adding different labels makes sense only if sssd_be has a different
>  label from the other sssd processes, but then you can simply have
>  selinux policies that label accordingly to the parent process in the
>  same directory, no? 
Yes, I tried to future-proof the directory structure against different
sssd processes that need access to the database (responders) but not the
keytabs, because I suspect we won't think about the directory structure
again once we set it up :-). And a single separate directory just seemed
easier to label with a new context than several files.

In the end, I don't care that much as long as the keytabs have different
contexts that the responders can't access.

>  
>  >     - Is there any reasonable fallback beyond re-fetching the keytab we
>  >       can do? I can't think of any..
>  
>  Be careful with refetches and so we do not want to end up in infinite
>  loops, where the keytab can be fetched but still operation fails for
>  whatever reason. 
Sure, I will add a more explicit note once we agree what the mechanism
would be.

>  One thing that should be checked is if the new keytab and old keytab
>  have the same keys in them, and, if so just stop re-fetching and raising
>  a fatal error (at list for a reasonable period of time). 
I thought about a flag in the ipa_ad_server_ctx structure, because after
the trust is deleted and set up again, the kvno is still 1. When you say
'keys', are you proposing to compare the secret key? (krb5_keytab_entry.key)

>  
>  >     - Should the IPA tools signal SSSD either via UNIX signals or via
>  >       the DBus API once a trust has been added or removed to minimize
>  >       the race condition and improve user experience? Currently I hope
>  >       not, the proposed mechanism should be enough but testing is still
>  >       required.
>  > 
>  > When the design page is agreed on, I'll turn the three subsections of
>  > the implementation phase into smaller tickets and close the tracker.
>  > 
>  > = Feature Name =
>  > 
>  > Related ticket(s):
>  >  * https://fedorahosted.org/sssd/ticket/2579
>  > 
>  > === Problem statement ===
>  > The next IPA release will support one-way trusts. SSSD needs to add support
>  > for this feature in its server mode.
>  > 
>  > === Use cases ===
>  > One-way trust to Active Directory where FreeIPA realm trusts Active Directory
>  > forest using cross-forest trust feature of AD but the AD forest does not trust
>  > FreeIPA realm. Users from AD forest can access resources in FreeIPA realm.
>  > 
>  > === Overview of the solution ===
>  > At a high level, SSSD needs to examine the trust objects whether they
>  > are one-way or two way trusts. For each one-way trust, SSSD needs to fetch
>  > and store the keytab and use the keytab to secure the connection. For
>  > two-way trusts, we can keep using the existing code that reuses the IPA
>  > realm and the system keytab for both IPA and AD connectins. Care must be
>  > taken to remove keytabs of trusts that were removed as well.
>  > 
>  > Fetching the keytab would be done by calling the `ipa-getkeytab` utility
>  > for every one-way trust. The keytab would only be (re)fetched if it's
>  > missing or if attempting to use the keytab failed. On the IPA server,
>  > we must make sure that the IPA server identity is allowed to read the keytab.
>  > 
>  > Because handling multiple keytabs increases the risk of failing connections in
>  > case the trust wasn't setup correctly, we need to modify the failover code to
>  > not set the whole back end offline in case connecting to a subdomain AD server
>  > fails. Instead, the subdomain will be marked as inactive for a short period of
>  > time, during which it would act as offline. The proper way of solving this
>  > problem would be to rework the failover module so that it can act per domain,
>  > not only per back end, however, that change is out of scope for this release.
>  > 
>  > === Implementation details ===
>  > This section describes all the required changes in detail.
>  > 
>  > ==== Reading the subdomains in the IPA subdomain handler ====
>  > The IPA subdomain handler will include the attribute `ipaNTTrustDirection`
>  > when reading the trust objects. Currently this attribute is not readable
>  > by the host principal, so the IPA ACIs must be relaxed (ipa ticket?). If
>  > the trust direction is set to an OR of `lsa.LSA_TRUST_DIRECTION_OUTBOUND`
>  > and `lsa.LSA_TRUST_DIRECTION_INBOUND`, then it's a two-way trust and we'll
>  > just use the existing code that re-uses the IPA keytab for the AD trusted
>  > domain as well. If the attribute is only `lsa.LSA_TRUST_DIRECTION_OUTBOUND`,
>  > we handle the trust as a one-way trust. The trust type can be stored in
>  > `ipa_ad_server_ctx`.
>  
>  What do you do if you have only lsa.LSA_TRUST_DIRECTION_INBOUND ? 
Log this object loudly and ignore. Is there another reasonable thing to
do other than skip such trust relationship ? (Yes, skipping the object
should be spelled out explicitly in the design page)

>  
>  > Each `sss_domain_info` structure will be created as `inactive` in the
>  > subdomain code. After enumerating the trusted domains, the subdomain
>  > handler will check if a keytab already exists for every one-way trusted
>  > domain. If yes, the domain is ready to use and can be enabled. If there is
>  > no keytab, the subdomain handler will fork out a call to `ipa-getkeytab`,
>  > fetch the keytab and store it under `/var/lib/sss/db`. The `ipa-getkeytab`
>  > call will be done using Kerberos credentials the host has. IPA ACIs must
>  > be modified accordingly to allow the IPA server principals to fetch the
>  > trust keytabs, but noone else.
>  > 
>  > The directory `/var/lib/sss/db` is already only accessible to the sssd
>  > user only. As an additional security measure, we might want to store the
>  > keytabs under a new directory so that the keytabs also receive a SELinux
>  > context stricter than the default `sssd_var_lib_t`. If fetching the keytab
>  > succeeds, the domain would be enabled.  The SELinux policy must also be
>  > adjusted to allow calling `ipa-getkeytab` by the `sssd_be` process.
>  
>  Getting a keytab is an LDAP extended operation, do you want to fork off
>  ipa-getkeytab to avoid coding the LDAP operatin in SSSD ? Or for some
>  other reason ? 
No, I just wanted to use code that is already there. Retrieving the
keytab should be rare enough to justify the overhead of forking an
executable and would let us to keep up with any changes IPA might do.

Or let me turn that question around -- do you see any advantage of
searching the keytab directly from SSSD?

>  
>  > If any trust relationships were removed, the corresponding keytabs must
>  > be removed from the disk as well.
>  
>  How do you do this ?
>  Just regularly enumerate the keytab files on disk and remove the ones
>  that have no object in the cache ? 
Yes, after the subdomain handler finishes. We alrady have a return boolean
'refresh_has_changes' and re-initialize domain-realm mapping for
example.

>  One advantage of using a separate subdir for keytabs, as you
> propose,
>  would be to wipe it clean every time sssd restart (at stop and at start
>  too for good measure), then enumerate the trusts on the server and
>  re-fetch any keytab. 
I thought about this, but it seemed a bit wasteful. But now that I think
of it, this would present a 'known control point' to start fresh..so it
might come handy.

>  
>  > ==== Changes to the AD id_ctx instantiation ====
>  > With two-way trust, we can keep using the default IPA principal and keytab.
>  > 
>  > With one-way trust, the keytab retrieved from the IPA server must be
>  > used. Also, the principal must be passed into the `ad_create_default_options`
>  > function. The custom values must be set before we proceed to instantiate
>  > LDAP provider options. The only AD provider option we need to set is
>  > AD_KRB5_REALM.
>  > 
>  > In the LDAP provider, we must take care that the following sdap options
>  > are set correctly:
>  >     * SDAP_SASL_AUTHID - must be set to the NetBIOS name of the IPA domain. (A
>  >       domain `TRUST.COM` would set this value to `TRUST$`).
>  
>  The NetBIOS domain can be completely unrelated to the DNS domain, so
>  make sure you do not just try to truncate the realm but actually fetch
>  the info from IPA/AD. 
Sure, sorry, that was meant as an example only. I added a more explicit
sentence.

>  
>  >     * SDAP_SASL_REALM - must be set to the AD realm
>  >     * SDAP_KRB5_KEYTAB - must be set to the per-domain keytab retrieved from IPA
>  > 
>  > The AD provider eventually calls `sdap_set_sasl_options()` from the
>  > LDAP provider, we need to make sure this function receives the correct
>  > values. During experimentation we were able to show that using multiple
>  > different SASL users and different realms doesn't cause any problems in
>  > SASL or LDAP libraries.
>  > 
>  > The only place that will keep using the IPA realm is the failover
>  > instantiation. We need to keep using this hack until failover is per-backend.
>  > 
>  > ==== Subdomain offline status changes ====
>  > At the moment, the whole back end can be either online or offline and
>  > the status applies to both the main domain and the subdomains alike. As
>  > an effect, a failure to connect to a subdomain server would also make the
>  > main domain operate offline. In many subdomain setups, it's actually more
>  > convenient not to, because the subdomain server might be on a different
>  > network segment, behind a different firewall etc. Instead, the domain
>  > should only be made inactive.
>  
>  Does this cover accurately the Forest vs AD subdomain cases ?
>  We will have a keytab only for the AD root domain (the forest) and we
>  need to use those credentials for any other domain in the AD forest. 
Well, both subdomains would be made inactive, then. The point is to
avoid marking the IPA domain as offline if the trusts break.

btw this would be also useful for AD-AD trusts because what I learned is
that AD subdomains tend to be isolated from one another on the network
level as well and then failure to reach other domains is currently fatal
which brings the whole sssd_be process offline.

>  
>  > The `sss_domain_info` structure would convert the 'bool disabled' parameter
>  > into an `enum sss_domain_state`. The supported values would be:
>  >     * ''disabled'' - the domain should not be used by either
> responder or
>  >       provider. It was removed or disabled on the server.
>  >     * ''active'' - the domain can be used by a responder and the
> data provider
>  >       would forward its request to the backend
>  >     * ''inactive'' - the domain can be used by a responder, but the
> data
>  >       provider would just shortcut as if the domain was offline. For now, this
>  >       option will be used by subdomains only.
>  > 
>  > The implementation would include renaming the existing `be_mark_offline()`
>  > function to be called `be_mark_dom_offline()` and modifying its behaviour. The
>  > existing code that sets the offline status and runs the offline callbacks
>  > would be called for parent domains only. For subdomains, we would mark the
>  > subdomain as inactive and schedule a tevent request that would unconditionally
>  > reset inactive domain to active. The request would be scheduled after
>  > `offline_timeout` seconds to be consistent with main domains from user's
>  > perspective. Likewise, the `be_reset_offline()` function will be extended to
>  > reset inactive domains to active as well as the SIGUSR1 and SIGUSR2 signal
>  > handlers. Finally, all calls to the `be_is_offline()` function should be
>  > inspected and the invocations that are per-domain should be converted to a new
>  > function `be_dom_is_offline()` that would check the offline status for parent
>  > domains and the offline state for subdomains. We should also make sure the
>  > backend offline status structure is opaque as currently its internals are
>  > readable by external users as well. Making the offline status opaque would
>  > make it safer to perform modifications to the offline code.
>  > 
>  > In both offline and inactive cases, the ID handlers would reply with
>  > `DP_ERR_OFFLINE`. The crucial difference between offline and inactive at this
>  > point would be that inactive domains are re-activated undonditionally. When we
>  > modify the failover code to handle domains separately, we'll be able to
>  > leverage per-domain online checks or online/offline callbacks as well.
>  > 
>  > ==== Detecting re-established trusts ====
>  > In cases the `sdap_kinit_send()` request fails, the sdap code would return a
>  > special error code instead of blindly returning `EIO` as it does at the
>  > moment. When the `ipa_get_ad_acct` request receives this error code, it would
>  > re-run the subdomain request in order to check if the trust relationship still
>  > exists and in order to re-fetch the keytab again. In case the trust is still
>  > established, we would re-set the inactive domain status and re-run the account
>  > request. In case the trust went away, the subdomain code should remove the
>  > trusted domain (however, this must be tested). If the account request returns
>  > any error after reinitializing the ketab, the subdomain would be marked
>  > inactive in order to avoid fetching the keytab again and again. In order to be
>  > able to run the subdomain request separately from the subdomain back end
>  > handler, the subdomain code must be wrapped into a subdomain request as the
>  > code currently assumes it's being called from the subdomain backend handler
>  > only.
>  > 
>  > === Future work ===
>  > Handling failover and offline status on per-domain basis instead of
>  > per-backend basis should be done in the next release.
>  > 
>  > === Configuration changes ===
>  > none
>  > 
>  > === How To Test ===
>  > Establish a one-way trust relationship with an AD domain. Make sure both
>  > IPA and AD users are resolvable. It's prudent to test combinations of
>  > one-way and two-way trusts with different forests. Make sure removing a
>  > trust relationship removes the keytab from the filesystem. Make sure that
>  > SSSD handles re-establishing a trust relationship.
>  > 
>  > === Authors ===
>  > * Jakub Hrozek <jhrozek(a)redhat.com&gt;
>  
>  Good job. 
Thank you for taking the time to review the design.

I made some changes to the design page, here is the diff:
https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts?action=diff&am...

I'll do one more change after we agree on how should we protect against
loops in fetching keytabs (flag vs. check keytab contents).

[Alexander Bokovoy]

On Mon, 27 Apr 2015, Jakub Hrozek wrote:
> On Sun, Apr 26, 2015 at 06:17:21PM -0400, Simo Sorce wrote:
> > Very nice writeup!
> > A few comments..
> >
> > On Sun, 2015-04-26 at 21:22 +0200, Jakub Hrozek wrote:
> > > Hi,
> > >
> > > I prepared design page for the one-way trusts feature:
> > >     https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts
> > >
> > > The text is copied below as well for your convenience. I have a couple
> > > of questions to discuss:
> > >     - Is it worth it to put the keytabs into a new subdirectory so that
> > >       a special SELinux label can be applied? I think yes, since then
> > >       we'd have additional layer of protection of the trust secrets
> >
> > It is not clear to me what would be the advantage, who are you trying to
> > protect against ?
> > Adding different labels makes sense only if sssd_be has a different
> > label from the other sssd processes, but then you can simply have
> > selinux policies that label accordingly to the parent process in the
> > same directory, no?
>
> Yes, I tried to future-proof the directory structure against different
> sssd processes that need access to the database (responders) but not the
> keytabs, because I suspect we won't think about the directory structure
> again once we set it up :-). And a single separate directory just seemed
> easier to label with a new context than several files.
>
> In the end, I don't care that much as long as the keytabs have different
> contexts that the responders can't access. 
Given that we want to have one keytab
per forest trust and naming
keytabs after forest trust name, we can live with a single directory. It
would be good, indeed, to have it separated from the rest of sssd.


> >
> > >     - Is there any reasonable fallback beyond re-fetching the keytab we
> > >       can do? I can't think of any..
> >
> > Be careful with refetches and so we do not want to end up in infinite
> > loops, where the keytab can be fetched but still operation fails for
> > whatever reason.
>
> Sure, I will add a more explicit note once we agree what the mechanism
> would be. 
Another thing to remember is a potential need to limit enctypes you'd
be
requesting because camellia ciphers are not know to AD and might cause
issues at some point.


> > One thing that should be checked is if the new keytab and old
> keytab
> > have the same keys in them, and, if so just stop re-fetching and raising
> > a fatal error (at list for a reasonable period of time).
>
> I thought about a flag in the ipa_ad_server_ctx structure, because after
> the trust is deleted and set up again, the kvno is still 1. When you say
> 'keys', are you proposing to compare the secret key? (krb5_keytab_entry.key)
Yes, don't use KVNO value to compare, verify actual keys.

> >
> > >     - Should the IPA tools signal SSSD either via UNIX signals or via
> > >       the DBus API once a trust has been added or removed to minimize
> > >       the race condition and improve user experience? Currently I hope
> > >       not, the proposed mechanism should be enough but testing is still
> > >       required.
> > >
> > > When the design page is agreed on, I'll turn the three subsections of
> > > the implementation phase into smaller tickets and close the tracker.
> > >
> > > = Feature Name =
> > >
> > > Related ticket(s):
> > >  * https://fedorahosted.org/sssd/ticket/2579
> > >
> > > === Problem statement ===
> > > The next IPA release will support one-way trusts. SSSD needs to add support
> > > for this feature in its server mode.
> > >
> > > === Use cases ===
> > > One-way trust to Active Directory where FreeIPA realm trusts Active Directory
> > > forest using cross-forest trust feature of AD but the AD forest does not trust
> > > FreeIPA realm. Users from AD forest can access resources in FreeIPA realm.
> > >
> > > === Overview of the solution ===
> > > At a high level, SSSD needs to examine the trust objects whether they
> > > are one-way or two way trusts. For each one-way trust, SSSD needs to fetch
> > > and store the keytab and use the keytab to secure the connection. For
> > > two-way trusts, we can keep using the existing code that reuses the IPA
> > > realm and the system keytab for both IPA and AD connectins. Care must be
> > > taken to remove keytabs of trusts that were removed as well.
> > >
> > > Fetching the keytab would be done by calling the `ipa-getkeytab` utility
> > > for every one-way trust. The keytab would only be (re)fetched if it's
> > > missing or if attempting to use the keytab failed. On the IPA server,
> > > we must make sure that the IPA server identity is allowed to read the keytab.
> > >
> > > Because handling multiple keytabs increases the risk of failing connections in
> > > case the trust wasn't setup correctly, we need to modify the failover code
> to
> > > not set the whole back end offline in case connecting to a subdomain AD server
> > > fails. Instead, the subdomain will be marked as inactive for a short period of
> > > time, during which it would act as offline. The proper way of solving this
> > > problem would be to rework the failover module so that it can act per domain,
> > > not only per back end, however, that change is out of scope for this release.
> > >
> > > === Implementation details ===
> > > This section describes all the required changes in detail.
> > >
> > > ==== Reading the subdomains in the IPA subdomain handler ====
> > > The IPA subdomain handler will include the attribute `ipaNTTrustDirection`
> > > when reading the trust objects. Currently this attribute is not readable
> > > by the host principal, so the IPA ACIs must be relaxed (ipa ticket?). If
> > > the trust direction is set to an OR of `lsa.LSA_TRUST_DIRECTION_OUTBOUND`
> > > and `lsa.LSA_TRUST_DIRECTION_INBOUND`, then it's a two-way trust and
> we'll
> > > just use the existing code that re-uses the IPA keytab for the AD trusted
> > > domain as well. If the attribute is only `lsa.LSA_TRUST_DIRECTION_OUTBOUND`,
> > > we handle the trust as a one-way trust. The trust type can be stored in
> > > `ipa_ad_server_ctx`.
> >
> > What do you do if you have only lsa.LSA_TRUST_DIRECTION_INBOUND ?
>
> Log this object loudly and ignore. Is there another reasonable thing to
> do other than skip such trust relationship ? (Yes, skipping the object
> should be spelled out explicitly in the design page) 
Yes, nothing else you can do.
I'm not sure at which point we'll get to
INBOUND-only case.


> > > Each `sss_domain_info` structure will be created as
> `inactive` in the
> > > subdomain code. After enumerating the trusted domains, the subdomain
> > > handler will check if a keytab already exists for every one-way trusted
> > > domain. If yes, the domain is ready to use and can be enabled. If there is
> > > no keytab, the subdomain handler will fork out a call to `ipa-getkeytab`,
> > > fetch the keytab and store it under `/var/lib/sss/db`. The `ipa-getkeytab`
> > > call will be done using Kerberos credentials the host has. IPA ACIs must
> > > be modified accordingly to allow the IPA server principals to fetch the
> > > trust keytabs, but noone else.
> > >
> > > The directory `/var/lib/sss/db` is already only accessible to the sssd
> > > user only. As an additional security measure, we might want to store the
> > > keytabs under a new directory so that the keytabs also receive a SELinux
> > > context stricter than the default `sssd_var_lib_t`. If fetching the keytab
> > > succeeds, the domain would be enabled.  The SELinux policy must also be
> > > adjusted to allow calling `ipa-getkeytab` by the `sssd_be` process.
> >
> > Getting a keytab is an LDAP extended operation, do you want to fork off
> > ipa-getkeytab to avoid coding the LDAP operatin in SSSD ? Or for some
> > other reason ?
>
> No, I just wanted to use code that is already there. Retrieving the
> keytab should be rare enough to justify the overhead of forking an
> executable and would let us to keep up with any changes IPA might do.
>
> Or let me turn that question around -- do you see any advantage of
> searching the keytab directly from SSSD? 
I can see a downside in the fact that
you'd need to handle ASN.1
encoding/decoding to fetch the keytab and if we ever change its
structure, SSSD would be another point of failure. Calling out to
ipa-getkeytab is giving us assurance to keep things working.


> > > If any trust relationships were removed, the corresponding
> keytabs must
> > > be removed from the disk as well.
> >
> > How do you do this ?
> > Just regularly enumerate the keytab files on disk and remove the ones
> > that have no object in the cache ?
>
> Yes, after the subdomain handler finishes. We alrady have a return boolean
> 'refresh_has_changes' and re-initialize domain-realm mapping for
> example.
>
> > One advantage of using a separate subdir for keytabs, as you propose,
> > would be to wipe it clean every time sssd restart (at stop and at start
> > too for good measure), then enumerate the trusts on the server and
> > re-fetch any keytab.
>
> I thought about this, but it seemed a bit wasteful. But now that I think
> of it, this would present a 'known control point' to start fresh..so it
> might come handy. 
If you do so, please make it triggered via some explicit
mechanism, not
as a default.
 
> > > ==== Changes to the AD id_ctx instantiation ====
> > > With two-way trust, we can keep using the default IPA principal and keytab.
> > >
> > > With one-way trust, the keytab retrieved from the IPA server must be
> > > used. Also, the principal must be passed into the `ad_create_default_options`
> > > function. The custom values must be set before we proceed to instantiate
> > > LDAP provider options. The only AD provider option we need to set is
> > > AD_KRB5_REALM.
> > >
> > > In the LDAP provider, we must take care that the following sdap options
> > > are set correctly:
> > >     * SDAP_SASL_AUTHID - must be set to the NetBIOS name of the IPA domain. (A
> > >       domain `TRUST.COM` would set this value to `TRUST$`).
> >
> > The NetBIOS domain can be completely unrelated to the DNS domain, so
> > make sure you do not just try to truncate the realm but actually fetch
> > the info from IPA/AD.
>
> Sure, sorry, that was meant as an example only. I added a more explicit
> sentence.
>
> >
> > >     * SDAP_SASL_REALM - must be set to the AD realm
> > >     * SDAP_KRB5_KEYTAB - must be set to the per-domain keytab retrieved from
> IPA
> > >
> > > The AD provider eventually calls `sdap_set_sasl_options()` from the
> > > LDAP provider, we need to make sure this function receives the correct
> > > values. During experimentation we were able to show that using multiple
> > > different SASL users and different realms doesn't cause any problems in
> > > SASL or LDAP libraries.
> > >
> > > The only place that will keep using the IPA realm is the failover
> > > instantiation. We need to keep using this hack until failover is per-backend.
> > >
> > > ==== Subdomain offline status changes ====
> > > At the moment, the whole back end can be either online or offline and
> > > the status applies to both the main domain and the subdomains alike. As
> > > an effect, a failure to connect to a subdomain server would also make the
> > > main domain operate offline. In many subdomain setups, it's actually more
> > > convenient not to, because the subdomain server might be on a different
> > > network segment, behind a different firewall etc. Instead, the domain
> > > should only be made inactive.
> >
> > Does this cover accurately the Forest vs AD subdomain cases ?
> > We will have a keytab only for the AD root domain (the forest) and we
> > need to use those credentials for any other domain in the AD forest.
>
> Well, both subdomains would be made inactive, then. The point is to
> avoid marking the IPA domain as offline if the trusts break. 
Yes, if you don't
have keytab with keys to TDO object, you cannot
authenticate against any of the forest's AD DCs, in any domain of the
forest, so all of them would be offline in this case. Our current
problem is that we offline IPA domain here instead of the subdomains.


> btw this would be also useful for AD-AD trusts because what I learned
> is
> that AD subdomains tend to be isolated from one another on the network
> level as well and then failure to reach other domains is currently fatal
> which brings the whole sssd_be process offline.
>
> >
> > > The `sss_domain_info` structure would convert the 'bool disabled'
> parameter
> > > into an `enum sss_domain_state`. The supported values would be:
> > >     * ''disabled'' - the domain should not be used by either
> responder or
> > >       provider. It was removed or disabled on the server.
> > >     * ''active'' - the domain can be used by a responder and the
> data provider
> > >       would forward its request to the backend
> > >     * ''inactive'' - the domain can be used by a responder, but
> the data
> > >       provider would just shortcut as if the domain was offline. For now, this
> > >       option will be used by subdomains only.
> > >
> > > The implementation would include renaming the existing `be_mark_offline()`
> > > function to be called `be_mark_dom_offline()` and modifying its behaviour. The
> > > existing code that sets the offline status and runs the offline callbacks
> > > would be called for parent domains only. For subdomains, we would mark the
> > > subdomain as inactive and schedule a tevent request that would unconditionally
> > > reset inactive domain to active. The request would be scheduled after
> > > `offline_timeout` seconds to be consistent with main domains from user's
> > > perspective. Likewise, the `be_reset_offline()` function will be extended to
> > > reset inactive domains to active as well as the SIGUSR1 and SIGUSR2 signal
> > > handlers. Finally, all calls to the `be_is_offline()` function should be
> > > inspected and the invocations that are per-domain should be converted to a new
> > > function `be_dom_is_offline()` that would check the offline status for parent
> > > domains and the offline state for subdomains. We should also make sure the
> > > backend offline status structure is opaque as currently its internals are
> > > readable by external users as well. Making the offline status opaque would
> > > make it safer to perform modifications to the offline code.
> > >
> > > In both offline and inactive cases, the ID handlers would reply with
> > > `DP_ERR_OFFLINE`. The crucial difference between offline and inactive at this
> > > point would be that inactive domains are re-activated undonditionally. When we
> > > modify the failover code to handle domains separately, we'll be able to
> > > leverage per-domain online checks or online/offline callbacks as well.
> > >
> > > ==== Detecting re-established trusts ====
> > > In cases the `sdap_kinit_send()` request fails, the sdap code would return a
> > > special error code instead of blindly returning `EIO` as it does at the
> > > moment. When the `ipa_get_ad_acct` request receives this error code, it would
> > > re-run the subdomain request in order to check if the trust relationship still
> > > exists and in order to re-fetch the keytab again. In case the trust is still
> > > established, we would re-set the inactive domain status and re-run the account
> > > request. In case the trust went away, the subdomain code should remove the
> > > trusted domain (however, this must be tested). If the account request returns
> > > any error after reinitializing the ketab, the subdomain would be marked
> > > inactive in order to avoid fetching the keytab again and again. In order to be
> > > able to run the subdomain request separately from the subdomain back end
> > > handler, the subdomain code must be wrapped into a subdomain request as the
> > > code currently assumes it's being called from the subdomain backend handler
> > > only.
> > >
> > > === Future work ===
> > > Handling failover and offline status on per-domain basis instead of
> > > per-backend basis should be done in the next release.
> > >
> > > === Configuration changes ===
> > > none
> > >
> > > === How To Test ===
> > > Establish a one-way trust relationship with an AD domain. Make sure both
> > > IPA and AD users are resolvable. It's prudent to test combinations of
> > > one-way and two-way trusts with different forests. Make sure removing a
> > > trust relationship removes the keytab from the filesystem. Make sure that
> > > SSSD handles re-establishing a trust relationship.
> > >
> > > === Authors ===
> > > * Jakub Hrozek <jhrozek(a)redhat.com&gt;
> >
> > Good job.
>
> Thank you for taking the time to review the design.
>
> I made some changes to the design page, here is the diff:
> https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts?action=diff&am...
>
> I'll do one more change after we agree on how should we protect against
> loops in fetching keytabs (flag vs. check keytab contents). 
I think the design is
sensible.
-- 
/ Alexander Bokovoy

[Simo Sorce]

On Mon, 2015-04-27 at 15:04 +0300, Alexander Bokovoy wrote:
>  On Mon, 27 Apr 2015, Jakub Hrozek wrote:
>  >On Sun, Apr 26, 2015 at 06:17:21PM -0400, Simo Sorce wrote:
>  >> Very nice writeup!
>  >> A few comments..
>  >>
>  >> On Sun, 2015-04-26 at 21:22 +0200, Jakub Hrozek wrote:
>  >> > Hi,
>  >> >
>  >> > I prepared design page for the one-way trusts feature:
>  >> >     https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts
>  >> >
>  >> > The text is copied below as well for your convenience. I have a couple
>  >> > of questions to discuss:
>  >> >     - Is it worth it to put the keytabs into a new subdirectory so that
>  >> >       a special SELinux label can be applied? I think yes, since then
>  >> >       we'd have additional layer of protection of the trust secrets
>  >>
>  >> It is not clear to me what would be the advantage, who are you trying to
>  >> protect against ?
>  >> Adding different labels makes sense only if sssd_be has a different
>  >> label from the other sssd processes, but then you can simply have
>  >> selinux policies that label accordingly to the parent process in the
>  >> same directory, no?
>  >
>  >Yes, I tried to future-proof the directory structure against different
>  >sssd processes that need access to the database (responders) but not the
>  >keytabs, because I suspect we won't think about the directory structure
>  >again once we set it up :-). And a single separate directory just seemed
>  >easier to label with a new context than several files.
>  >
>  >In the end, I don't care that much as long as the keytabs have different
>  >contexts that the responders can't access.
>  Given that we want to have one keytab per forest trust and naming
>  keytabs after forest trust name, we can live with a single directory. It
>  would be good, indeed, to have it separated from the rest of sssd.
>  
>  
>  >>
>  >> >     - Is there any reasonable fallback beyond re-fetching the keytab we
>  >> >       can do? I can't think of any..
>  >>
>  >> Be careful with refetches and so we do not want to end up in infinite
>  >> loops, where the keytab can be fetched but still operation fails for
>  >> whatever reason.
>  >
>  >Sure, I will add a more explicit note once we agree what the mechanism
>  >would be.
>  Another thing to remember is a potential need to limit enctypes you'd be
>  requesting because camellia ciphers are not know to AD and might cause
>  issues at some point. 
SSSD should not be in the business of *creating* keytabs, it should only
be allowed to retrieve a precreated key, so SSSD shouldn't care about
enctypes, it will get only those that the FreeIPA code stored in the key
in LDAP.

>  >> One thing that should be checked is if the new keytab and
> old keytab
>  >> have the same keys in them, and, if so just stop re-fetching and raising
>  >> a fatal error (at list for a reasonable period of time).
>  >
>  >I thought about a flag in the ipa_ad_server_ctx structure, because after
>  >the trust is deleted and set up again, the kvno is still 1. When you say
>  >'keys', are you proposing to compare the secret key? (krb5_keytab_entry.key)
>  Yes, don't use KVNO value to compare, verify actual keys. 
It would be nice to avoid the need to link directly to krb5 calls, but
it is probably safer to check by key value.

>  >>
>  >> >     - Should the IPA tools signal SSSD either via UNIX signals or via
>  >> >       the DBus API once a trust has been added or removed to minimize
>  >> >       the race condition and improve user experience? Currently I hope
>  >> >       not, the proposed mechanism should be enough but testing is still
>  >> >       required.
>  >> >
>  >> > When the design page is agreed on, I'll turn the three subsections of
>  >> > the implementation phase into smaller tickets and close the tracker.
>  >> >
>  >> > = Feature Name =
>  >> >
>  >> > Related ticket(s):
>  >> >  * https://fedorahosted.org/sssd/ticket/2579
>  >> >
>  >> > === Problem statement ===
>  >> > The next IPA release will support one-way trusts. SSSD needs to add
> support
>  >> > for this feature in its server mode.
>  >> >
>  >> > === Use cases ===
>  >> > One-way trust to Active Directory where FreeIPA realm trusts Active
> Directory
>  >> > forest using cross-forest trust feature of AD but the AD forest does not
> trust
>  >> > FreeIPA realm. Users from AD forest can access resources in FreeIPA realm.
>  >> >
>  >> > === Overview of the solution ===
>  >> > At a high level, SSSD needs to examine the trust objects whether they
>  >> > are one-way or two way trusts. For each one-way trust, SSSD needs to fetch
>  >> > and store the keytab and use the keytab to secure the connection. For
>  >> > two-way trusts, we can keep using the existing code that reuses the IPA
>  >> > realm and the system keytab for both IPA and AD connectins. Care must be
>  >> > taken to remove keytabs of trusts that were removed as well.
>  >> >
>  >> > Fetching the keytab would be done by calling the `ipa-getkeytab` utility
>  >> > for every one-way trust. The keytab would only be (re)fetched if it's
>  >> > missing or if attempting to use the keytab failed. On the IPA server,
>  >> > we must make sure that the IPA server identity is allowed to read the
> keytab.
>  >> >
>  >> > Because handling multiple keytabs increases the risk of failing connections
> in
>  >> > case the trust wasn't setup correctly, we need to modify the failover
> code to
>  >> > not set the whole back end offline in case connecting to a subdomain AD
> server
>  >> > fails. Instead, the subdomain will be marked as inactive for a short period
> of
>  >> > time, during which it would act as offline. The proper way of solving this
>  >> > problem would be to rework the failover module so that it can act per
> domain,
>  >> > not only per back end, however, that change is out of scope for this
> release.
>  >> >
>  >> > === Implementation details ===
>  >> > This section describes all the required changes in detail.
>  >> >
>  >> > ==== Reading the subdomains in the IPA subdomain handler ====
>  >> > The IPA subdomain handler will include the attribute `ipaNTTrustDirection`
>  >> > when reading the trust objects. Currently this attribute is not readable
>  >> > by the host principal, so the IPA ACIs must be relaxed (ipa ticket?). If
>  >> > the trust direction is set to an OR of `lsa.LSA_TRUST_DIRECTION_OUTBOUND`
>  >> > and `lsa.LSA_TRUST_DIRECTION_INBOUND`, then it's a two-way trust and
> we'll
>  >> > just use the existing code that re-uses the IPA keytab for the AD trusted
>  >> > domain as well. If the attribute is only
> `lsa.LSA_TRUST_DIRECTION_OUTBOUND`,
>  >> > we handle the trust as a one-way trust. The trust type can be stored in
>  >> > `ipa_ad_server_ctx`.
>  >>
>  >> What do you do if you have only lsa.LSA_TRUST_DIRECTION_INBOUND ?
>  >
>  >Log this object loudly and ignore. Is there another reasonable thing to
>  >do other than skip such trust relationship ? (Yes, skipping the object
>  >should be spelled out explicitly in the design page)
>  Yes, nothing else you can do. I'm not sure at which point we'll get to
>  INBOUND-only case. 
We should probably just ignore w/o even screaming too loud.

>  >> > Each `sss_domain_info` structure will be created as
> `inactive` in the
>  >> > subdomain code. After enumerating the trusted domains, the subdomain
>  >> > handler will check if a keytab already exists for every one-way trusted
>  >> > domain. If yes, the domain is ready to use and can be enabled. If there is
>  >> > no keytab, the subdomain handler will fork out a call to `ipa-getkeytab`,
>  >> > fetch the keytab and store it under `/var/lib/sss/db`. The `ipa-getkeytab`
>  >> > call will be done using Kerberos credentials the host has. IPA ACIs must
>  >> > be modified accordingly to allow the IPA server principals to fetch the
>  >> > trust keytabs, but noone else.
>  >> >
>  >> > The directory `/var/lib/sss/db` is already only accessible to the sssd
>  >> > user only. As an additional security measure, we might want to store the
>  >> > keytabs under a new directory so that the keytabs also receive a SELinux
>  >> > context stricter than the default `sssd_var_lib_t`. If fetching the keytab
>  >> > succeeds, the domain would be enabled.  The SELinux policy must also be
>  >> > adjusted to allow calling `ipa-getkeytab` by the `sssd_be` process.
>  >>
>  >> Getting a keytab is an LDAP extended operation, do you want to fork off
>  >> ipa-getkeytab to avoid coding the LDAP operatin in SSSD ? Or for some
>  >> other reason ?
>  >
>  >No, I just wanted to use code that is already there. Retrieving the
>  >keytab should be rare enough to justify the overhead of forking an
>  >executable and would let us to keep up with any changes IPA might do.
>  >
>  >Or let me turn that question around -- do you see any advantage of
>  >searching the keytab directly from SSSD?
>  I can see a downside in the fact that you'd need to handle ASN.1
>  encoding/decoding to fetch the keytab and if we ever change its
>  structure, SSSD would be another point of failure. Calling out to
>  ipa-getkeytab is giving us assurance to keep things working. 
We cannot change the format, if we do we'll have to indtroduce a new
version and support the old one. So this is not a problem.

The advantage of doing this directly is if we decide (for whatever
reason) to store the key material directly in the ldb cache instead of a
file. If we did that then we could use a MEMORY keytab for the various
operations by fetching it out of the cache and it would not ever hit the
filesystem. Hoewever we may not want to have that key material in LDB at
all, it is just a possibility.

>  >> > If any trust relationships were removed, the
> corresponding keytabs must
>  >> > be removed from the disk as well.
>  >>
>  >> How do you do this ?
>  >> Just regularly enumerate the keytab files on disk and remove the ones
>  >> that have no object in the cache ?
>  >
>  >Yes, after the subdomain handler finishes. We alrady have a return boolean
>  >'refresh_has_changes' and re-initialize domain-realm mapping for
>  >example.
>  >
>  >> One advantage of using a separate subdir for keytabs, as you propose,
>  >> would be to wipe it clean every time sssd restart (at stop and at start
>  >> too for good measure), then enumerate the trusts on the server and
>  >> re-fetch any keytab.
>  >
>  >I thought about this, but it seemed a bit wasteful. But now that I think
>  >of it, this would present a 'known control point' to start fresh..so it
>  >might come handy.
>  If you do so, please make it triggered via some explicit mechanism, not
>  as a default. 
Why not do it at restart ?

>  >> > ==== Changes to the AD id_ctx instantiation ====
>  >> > With two-way trust, we can keep using the default IPA principal and
> keytab.
>  >> >
>  >> > With one-way trust, the keytab retrieved from the IPA server must be
>  >> > used. Also, the principal must be passed into the
> `ad_create_default_options`
>  >> > function. The custom values must be set before we proceed to instantiate
>  >> > LDAP provider options. The only AD provider option we need to set is
>  >> > AD_KRB5_REALM.
>  >> >
>  >> > In the LDAP provider, we must take care that the following sdap options
>  >> > are set correctly:
>  >> >     * SDAP_SASL_AUTHID - must be set to the NetBIOS name of the IPA domain.
> (A
>  >> >       domain `TRUST.COM` would set this value to `TRUST$`).
>  >>
>  >> The NetBIOS domain can be completely unrelated to the DNS domain, so
>  >> make sure you do not just try to truncate the realm but actually fetch
>  >> the info from IPA/AD.
>  >
>  >Sure, sorry, that was meant as an example only. I added a more explicit
>  >sentence.
>  >
>  >>
>  >> >     * SDAP_SASL_REALM - must be set to the AD realm
>  >> >     * SDAP_KRB5_KEYTAB - must be set to the per-domain keytab retrieved
> from IPA
>  >> >
>  >> > The AD provider eventually calls `sdap_set_sasl_options()` from the
>  >> > LDAP provider, we need to make sure this function receives the correct
>  >> > values. During experimentation we were able to show that using multiple
>  >> > different SASL users and different realms doesn't cause any problems
> in
>  >> > SASL or LDAP libraries.
>  >> >
>  >> > The only place that will keep using the IPA realm is the failover
>  >> > instantiation. We need to keep using this hack until failover is
> per-backend.
>  >> >
>  >> > ==== Subdomain offline status changes ====
>  >> > At the moment, the whole back end can be either online or offline and
>  >> > the status applies to both the main domain and the subdomains alike. As
>  >> > an effect, a failure to connect to a subdomain server would also make the
>  >> > main domain operate offline. In many subdomain setups, it's actually
> more
>  >> > convenient not to, because the subdomain server might be on a different
>  >> > network segment, behind a different firewall etc. Instead, the domain
>  >> > should only be made inactive.
>  >>
>  >> Does this cover accurately the Forest vs AD subdomain cases ?
>  >> We will have a keytab only for the AD root domain (the forest) and we
>  >> need to use those credentials for any other domain in the AD forest.
>  >
>  >Well, both subdomains would be made inactive, then. The point is to
>  >avoid marking the IPA domain as offline if the trusts break.
>  Yes, if you don't have keytab with keys to TDO object, you cannot
>  authenticate against any of the forest's AD DCs, in any domain of the
>  forest, so all of them would be offline in this case. Our current
>  problem is that we offline IPA domain here instead of the subdomains.
>  
>  
>  >btw this would be also useful for AD-AD trusts because what I learned is
>  >that AD subdomains tend to be isolated from one another on the network
>  >level as well and then failure to reach other domains is currently fatal
>  >which brings the whole sssd_be process offline.
>  >
>  >>
>  >> > The `sss_domain_info` structure would convert the 'bool disabled'
> parameter
>  >> > into an `enum sss_domain_state`. The supported values would be:
>  >> >     * ''disabled'' - the domain should not be used by
> either responder or
>  >> >       provider. It was removed or disabled on the server.
>  >> >     * ''active'' - the domain can be used by a responder
> and the data provider
>  >> >       would forward its request to the backend
>  >> >     * ''inactive'' - the domain can be used by a responder,
> but the data
>  >> >       provider would just shortcut as if the domain was offline. For now,
> this
>  >> >       option will be used by subdomains only.
>  >> >
>  >> > The implementation would include renaming the existing `be_mark_offline()`
>  >> > function to be called `be_mark_dom_offline()` and modifying its behaviour.
> The
>  >> > existing code that sets the offline status and runs the offline callbacks
>  >> > would be called for parent domains only. For subdomains, we would mark the
>  >> > subdomain as inactive and schedule a tevent request that would
> unconditionally
>  >> > reset inactive domain to active. The request would be scheduled after
>  >> > `offline_timeout` seconds to be consistent with main domains from
> user's
>  >> > perspective. Likewise, the `be_reset_offline()` function will be extended
> to
>  >> > reset inactive domains to active as well as the SIGUSR1 and SIGUSR2 signal
>  >> > handlers. Finally, all calls to the `be_is_offline()` function should be
>  >> > inspected and the invocations that are per-domain should be converted to a
> new
>  >> > function `be_dom_is_offline()` that would check the offline status for
> parent
>  >> > domains and the offline state for subdomains. We should also make sure the
>  >> > backend offline status structure is opaque as currently its internals are
>  >> > readable by external users as well. Making the offline status opaque would
>  >> > make it safer to perform modifications to the offline code.
>  >> >
>  >> > In both offline and inactive cases, the ID handlers would reply with
>  >> > `DP_ERR_OFFLINE`. The crucial difference between offline and inactive at
> this
>  >> > point would be that inactive domains are re-activated undonditionally. When
> we
>  >> > modify the failover code to handle domains separately, we'll be able
> to
>  >> > leverage per-domain online checks or online/offline callbacks as well.
>  >> >
>  >> > ==== Detecting re-established trusts ====
>  >> > In cases the `sdap_kinit_send()` request fails, the sdap code would return
> a
>  >> > special error code instead of blindly returning `EIO` as it does at the
>  >> > moment. When the `ipa_get_ad_acct` request receives this error code, it
> would
>  >> > re-run the subdomain request in order to check if the trust relationship
> still
>  >> > exists and in order to re-fetch the keytab again. In case the trust is
> still
>  >> > established, we would re-set the inactive domain status and re-run the
> account
>  >> > request. In case the trust went away, the subdomain code should remove the
>  >> > trusted domain (however, this must be tested). If the account request
> returns
>  >> > any error after reinitializing the ketab, the subdomain would be marked
>  >> > inactive in order to avoid fetching the keytab again and again. In order to
> be
>  >> > able to run the subdomain request separately from the subdomain back end
>  >> > handler, the subdomain code must be wrapped into a subdomain request as
> the
>  >> > code currently assumes it's being called from the subdomain backend
> handler
>  >> > only.
>  >> >
>  >> > === Future work ===
>  >> > Handling failover and offline status on per-domain basis instead of
>  >> > per-backend basis should be done in the next release.
>  >> >
>  >> > === Configuration changes ===
>  >> > none
>  >> >
>  >> > === How To Test ===
>  >> > Establish a one-way trust relationship with an AD domain. Make sure both
>  >> > IPA and AD users are resolvable. It's prudent to test combinations of
>  >> > one-way and two-way trusts with different forests. Make sure removing a
>  >> > trust relationship removes the keytab from the filesystem. Make sure that
>  >> > SSSD handles re-establishing a trust relationship.
>  >> >
>  >> > === Authors ===
>  >> > * Jakub Hrozek <jhrozek(a)redhat.com&gt;
>  >>
>  >> Good job.
>  >
>  >Thank you for taking the time to review the design.
>  >
>  >I made some changes to the design page, here is the diff:
>
> >https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts?action=diff&version=9&old_version=8
>  >
>  >I'll do one more change after we agree on how should we protect against
>  >loops in fetching keytabs (flag vs. check keytab contents).
>  I think the design is sensible. 
Ack.

Simo.

[Alexander Bokovoy]

On Mon, 27 Apr 2015, Simo Sorce wrote:
> On Mon, 2015-04-27 at 15:04 +0300, Alexander Bokovoy wrote:
> > On Mon, 27 Apr 2015, Jakub Hrozek wrote:
> > >On Sun, Apr 26, 2015 at 06:17:21PM -0400, Simo Sorce wrote:
> > >> Very nice writeup!
> > >> A few comments..
> > >>
> > >> On Sun, 2015-04-26 at 21:22 +0200, Jakub Hrozek wrote:
> > >> > Hi,
> > >> >
> > >> > I prepared design page for the one-way trusts feature:
> > >> >     https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts
> > >> >
> > >> > The text is copied below as well for your convenience. I have a couple
> > >> > of questions to discuss:
> > >> >     - Is it worth it to put the keytabs into a new subdirectory so
> that
> > >> >       a special SELinux label can be applied? I think yes, since then
> > >> >       we'd have additional layer of protection of the trust
> secrets
> > >>
> > >> It is not clear to me what would be the advantage, who are you trying to
> > >> protect against ?
> > >> Adding different labels makes sense only if sssd_be has a different
> > >> label from the other sssd processes, but then you can simply have
> > >> selinux policies that label accordingly to the parent process in the
> > >> same directory, no?
> > >
> > >Yes, I tried to future-proof the directory structure against different
> > >sssd processes that need access to the database (responders) but not the
> > >keytabs, because I suspect we won't think about the directory structure
> > >again once we set it up :-). And a single separate directory just seemed
> > >easier to label with a new context than several files.
> > >
> > >In the end, I don't care that much as long as the keytabs have different
> > >contexts that the responders can't access.
> > Given that we want to have one keytab per forest trust and naming
> > keytabs after forest trust name, we can live with a single directory. It
> > would be good, indeed, to have it separated from the rest of sssd.
> >
> >
> > >>
> > >> >     - Is there any reasonable fallback beyond re-fetching the keytab
> we
> > >> >       can do? I can't think of any..
> > >>
> > >> Be careful with refetches and so we do not want to end up in infinite
> > >> loops, where the keytab can be fetched but still operation fails for
> > >> whatever reason.
> > >
> > >Sure, I will add a more explicit note once we agree what the mechanism
> > >would be.
> > Another thing to remember is a potential need to limit enctypes you'd be
> > requesting because camellia ciphers are not know to AD and might cause
> > issues at some point.
>
> SSSD should not be in the business of *creating* keytabs, it should only
> be allowed to retrieve a precreated key, so SSSD shouldn't care about
> enctypes, it will get only those that the FreeIPA code stored in the key
> in LDAP. 
So this means ipasam needs to limit enctypes when asking for the keys.


> > >> One thing that should be checked is if the new keytab
> and old keytab
> > >> have the same keys in them, and, if so just stop re-fetching and raising
> > >> a fatal error (at list for a reasonable period of time).
> > >
> > >I thought about a flag in the ipa_ad_server_ctx structure, because after
> > >the trust is deleted and set up again, the kvno is still 1. When you say
> > >'keys', are you proposing to compare the secret key?
> (krb5_keytab_entry.key)
> > Yes, don't use KVNO value to compare, verify actual keys.
>
> It would be nice to avoid the need to link directly to krb5 calls, but
> it is probably safer to check by key value. 
Note that we are dealing with a key
material from a separate realm and
we create the key in IPA LDAP by means of ipasam which results in the
fact there is no way to co-relate KVNO between the key in IPA LDAP and in AD
LDAP other than by authenticating or comparing key value to the local
copy.

On the other hand, when trust is re-established, we actually remove it
completely and re-create the objects. This means for new trust KVNO will
always be 1 and will only increase if there was a TDO password change
operation initiated by AD side (which currently doesn't work, we have
this part broken). So I think we probably need to remember a timestamp
of when trust object -- not the Kerberos principal but the trust object
that ipasam maintains for Samba -- was created/modified and compare that
with the time we requested keytab before as a short circuit. As we run
on the same server as LDAP server, that time should be comparable.

> > >> >     - Should the IPA tools signal SSSD either via
> UNIX signals or via
> > >> >       the DBus API once a trust has been added or removed to minimize
> > >> >       the race condition and improve user experience? Currently I hope
> > >> >       not, the proposed mechanism should be enough but testing is
> still
> > >> >       required.
> > >> >
> > >> > When the design page is agreed on, I'll turn the three subsections
> of
> > >> > the implementation phase into smaller tickets and close the tracker.
> > >> >
> > >> > = Feature Name =
> > >> >
> > >> > Related ticket(s):
> > >> >  * https://fedorahosted.org/sssd/ticket/2579
> > >> >
> > >> > === Problem statement ===
> > >> > The next IPA release will support one-way trusts. SSSD needs to add
> support
> > >> > for this feature in its server mode.
> > >> >
> > >> > === Use cases ===
> > >> > One-way trust to Active Directory where FreeIPA realm trusts Active
> Directory
> > >> > forest using cross-forest trust feature of AD but the AD forest does
> not trust
> > >> > FreeIPA realm. Users from AD forest can access resources in FreeIPA
> realm.
> > >> >
> > >> > === Overview of the solution ===
> > >> > At a high level, SSSD needs to examine the trust objects whether they
> > >> > are one-way or two way trusts. For each one-way trust, SSSD needs to
> fetch
> > >> > and store the keytab and use the keytab to secure the connection. For
> > >> > two-way trusts, we can keep using the existing code that reuses the
> IPA
> > >> > realm and the system keytab for both IPA and AD connectins. Care must
> be
> > >> > taken to remove keytabs of trusts that were removed as well.
> > >> >
> > >> > Fetching the keytab would be done by calling the `ipa-getkeytab`
> utility
> > >> > for every one-way trust. The keytab would only be (re)fetched if
> it's
> > >> > missing or if attempting to use the keytab failed. On the IPA server,
> > >> > we must make sure that the IPA server identity is allowed to read the
> keytab.
> > >> >
> > >> > Because handling multiple keytabs increases the risk of failing
> connections in
> > >> > case the trust wasn't setup correctly, we need to modify the
> failover code to
> > >> > not set the whole back end offline in case connecting to a subdomain AD
> server
> > >> > fails. Instead, the subdomain will be marked as inactive for a short
> period of
> > >> > time, during which it would act as offline. The proper way of solving
> this
> > >> > problem would be to rework the failover module so that it can act per
> domain,
> > >> > not only per back end, however, that change is out of scope for this
> release.
> > >> >
> > >> > === Implementation details ===
> > >> > This section describes all the required changes in detail.
> > >> >
> > >> > ==== Reading the subdomains in the IPA subdomain handler ====
> > >> > The IPA subdomain handler will include the attribute
> `ipaNTTrustDirection`
> > >> > when reading the trust objects. Currently this attribute is not
> readable
> > >> > by the host principal, so the IPA ACIs must be relaxed (ipa ticket?).
> If
> > >> > the trust direction is set to an OR of
> `lsa.LSA_TRUST_DIRECTION_OUTBOUND`
> > >> > and `lsa.LSA_TRUST_DIRECTION_INBOUND`, then it's a two-way trust
> and we'll
> > >> > just use the existing code that re-uses the IPA keytab for the AD
> trusted
> > >> > domain as well. If the attribute is only
> `lsa.LSA_TRUST_DIRECTION_OUTBOUND`,
> > >> > we handle the trust as a one-way trust. The trust type can be stored
> in
> > >> > `ipa_ad_server_ctx`.
> > >>
> > >> What do you do if you have only lsa.LSA_TRUST_DIRECTION_INBOUND ?
> > >
> > >Log this object loudly and ignore. Is there another reasonable thing to
> > >do other than skip such trust relationship ? (Yes, skipping the object
> > >should be spelled out explicitly in the design page)
> > Yes, nothing else you can do. I'm not sure at which point we'll get to
> > INBOUND-only case.
>
> We should probably just ignore w/o even screaming too loud. 
Yep.

> > >> > Each `sss_domain_info` structure will be created as `inactive` in the
> > >> > subdomain code. After enumerating the trusted domains, the subdomain
> > >> > handler will check if a keytab already exists for every one-way
> trusted
> > >> > domain. If yes, the domain is ready to use and can be enabled. If there
> is
> > >> > no keytab, the subdomain handler will fork out a call to
> `ipa-getkeytab`,
> > >> > fetch the keytab and store it under `/var/lib/sss/db`. The
> `ipa-getkeytab`
> > >> > call will be done using Kerberos credentials the host has. IPA ACIs
> must
> > >> > be modified accordingly to allow the IPA server principals to fetch
> the
> > >> > trust keytabs, but noone else.
> > >> >
> > >> > The directory `/var/lib/sss/db` is already only accessible to the sssd
> > >> > user only. As an additional security measure, we might want to store
> the
> > >> > keytabs under a new directory so that the keytabs also receive a
> SELinux
> > >> > context stricter than the default `sssd_var_lib_t`. If fetching the
> keytab
> > >> > succeeds, the domain would be enabled.  The SELinux policy must also
> be
> > >> > adjusted to allow calling `ipa-getkeytab` by the `sssd_be` process.
> > >>
> > >> Getting a keytab is an LDAP extended operation, do you want to fork off
> > >> ipa-getkeytab to avoid coding the LDAP operatin in SSSD ? Or for some
> > >> other reason ?
> > >
> > >No, I just wanted to use code that is already there. Retrieving the
> > >keytab should be rare enough to justify the overhead of forking an
> > >executable and would let us to keep up with any changes IPA might do.
> > >
> > >Or let me turn that question around -- do you see any advantage of
> > >searching the keytab directly from SSSD?
> > I can see a downside in the fact that you'd need to handle ASN.1
> > encoding/decoding to fetch the keytab and if we ever change its
> > structure, SSSD would be another point of failure. Calling out to
> > ipa-getkeytab is giving us assurance to keep things working.
>
> We cannot change the format, if we do we'll have to indtroduce a new
> version and support the old one. So this is not a problem.
>
> The advantage of doing this directly is if we decide (for whatever
> reason) to store the key material directly in the ldb cache instead of a
> file. If we did that then we could use a MEMORY keytab for the various
> operations by fetching it out of the cache and it would not ever hit the
> filesystem. Hoewever we may not want to have that key material in LDB at
> all, it is just a possibility. 
Right, it is currently easier to call out and leave
ipa-getkeytab to do
the job because we are practically running on IPA master in this more.


> > >> > If any trust relationships were removed, the
> corresponding keytabs must
> > >> > be removed from the disk as well.
> > >>
> > >> How do you do this ?
> > >> Just regularly enumerate the keytab files on disk and remove the ones
> > >> that have no object in the cache ?
> > >
> > >Yes, after the subdomain handler finishes. We alrady have a return boolean
> > >'refresh_has_changes' and re-initialize domain-realm mapping for
> > >example.
> > >
> > >> One advantage of using a separate subdir for keytabs, as you propose,
> > >> would be to wipe it clean every time sssd restart (at stop and at start
> > >> too for good measure), then enumerate the trusts on the server and
> > >> re-fetch any keytab.
> > >
> > >I thought about this, but it seemed a bit wasteful. But now that I think
> > >of it, this would present a 'known control point' to start fresh..so it
> > >might come handy.
> > If you do so, please make it triggered via some explicit mechanism, not
> > as a default.
>
> Why not do it at restart ? 
Do you want to always re-retrive the keys? I was under
assumption we are
not going to do that.


> > >> > ==== Changes to the AD id_ctx instantiation ====
> > >> > With two-way trust, we can keep using the default IPA principal and
> keytab.
> > >> >
> > >> > With one-way trust, the keytab retrieved from the IPA server must be
> > >> > used. Also, the principal must be passed into the
> `ad_create_default_options`
> > >> > function. The custom values must be set before we proceed to
> instantiate
> > >> > LDAP provider options. The only AD provider option we need to set is
> > >> > AD_KRB5_REALM.
> > >> >
> > >> > In the LDAP provider, we must take care that the following sdap
> options
> > >> > are set correctly:
> > >> >     * SDAP_SASL_AUTHID - must be set to the NetBIOS name of the IPA
> domain. (A
> > >> >       domain `TRUST.COM` would set this value to `TRUST$`).
> > >>
> > >> The NetBIOS domain can be completely unrelated to the DNS domain, so
> > >> make sure you do not just try to truncate the realm but actually fetch
> > >> the info from IPA/AD.
> > >
> > >Sure, sorry, that was meant as an example only. I added a more explicit
> > >sentence.
> > >
> > >>
> > >> >     * SDAP_SASL_REALM - must be set to the AD realm
> > >> >     * SDAP_KRB5_KEYTAB - must be set to the per-domain keytab retrieved
> from IPA
> > >> >
> > >> > The AD provider eventually calls `sdap_set_sasl_options()` from the
> > >> > LDAP provider, we need to make sure this function receives the correct
> > >> > values. During experimentation we were able to show that using
> multiple
> > >> > different SASL users and different realms doesn't cause any
> problems in
> > >> > SASL or LDAP libraries.
> > >> >
> > >> > The only place that will keep using the IPA realm is the failover
> > >> > instantiation. We need to keep using this hack until failover is
> per-backend.
> > >> >
> > >> > ==== Subdomain offline status changes ====
> > >> > At the moment, the whole back end can be either online or offline and
> > >> > the status applies to both the main domain and the subdomains alike.
> As
> > >> > an effect, a failure to connect to a subdomain server would also make
> the
> > >> > main domain operate offline. In many subdomain setups, it's
> actually more
> > >> > convenient not to, because the subdomain server might be on a
> different
> > >> > network segment, behind a different firewall etc. Instead, the domain
> > >> > should only be made inactive.
> > >>
> > >> Does this cover accurately the Forest vs AD subdomain cases ?
> > >> We will have a keytab only for the AD root domain (the forest) and we
> > >> need to use those credentials for any other domain in the AD forest.
> > >
> > >Well, both subdomains would be made inactive, then. The point is to
> > >avoid marking the IPA domain as offline if the trusts break.
> > Yes, if you don't have keytab with keys to TDO object, you cannot
> > authenticate against any of the forest's AD DCs, in any domain of the
> > forest, so all of them would be offline in this case. Our current
> > problem is that we offline IPA domain here instead of the subdomains.
> >
> >
> > >btw this would be also useful for AD-AD trusts because what I learned is
> > >that AD subdomains tend to be isolated from one another on the network
> > >level as well and then failure to reach other domains is currently fatal
> > >which brings the whole sssd_be process offline.
> > >
> > >>
> > >> > The `sss_domain_info` structure would convert the 'bool
> disabled' parameter
> > >> > into an `enum sss_domain_state`. The supported values would be:
> > >> >     * ''disabled'' - the domain should not be used by
> either responder or
> > >> >       provider. It was removed or disabled on the server.
> > >> >     * ''active'' - the domain can be used by a
> responder and the data provider
> > >> >       would forward its request to the backend
> > >> >     * ''inactive'' - the domain can be used by a
> responder, but the data
> > >> >       provider would just shortcut as if the domain was offline. For
> now, this
> > >> >       option will be used by subdomains only.
> > >> >
> > >> > The implementation would include renaming the existing
> `be_mark_offline()`
> > >> > function to be called `be_mark_dom_offline()` and modifying its
> behaviour. The
> > >> > existing code that sets the offline status and runs the offline
> callbacks
> > >> > would be called for parent domains only. For subdomains, we would mark
> the
> > >> > subdomain as inactive and schedule a tevent request that would
> unconditionally
> > >> > reset inactive domain to active. The request would be scheduled after
> > >> > `offline_timeout` seconds to be consistent with main domains from
> user's
> > >> > perspective. Likewise, the `be_reset_offline()` function will be
> extended to
> > >> > reset inactive domains to active as well as the SIGUSR1 and SIGUSR2
> signal
> > >> > handlers. Finally, all calls to the `be_is_offline()` function should
> be
> > >> > inspected and the invocations that are per-domain should be converted
> to a new
> > >> > function `be_dom_is_offline()` that would check the offline status for
> parent
> > >> > domains and the offline state for subdomains. We should also make sure
> the
> > >> > backend offline status structure is opaque as currently its internals
> are
> > >> > readable by external users as well. Making the offline status opaque
> would
> > >> > make it safer to perform modifications to the offline code.
> > >> >
> > >> > In both offline and inactive cases, the ID handlers would reply with
> > >> > `DP_ERR_OFFLINE`. The crucial difference between offline and inactive
> at this
> > >> > point would be that inactive domains are re-activated undonditionally.
> When we
> > >> > modify the failover code to handle domains separately, we'll be
> able to
> > >> > leverage per-domain online checks or online/offline callbacks as well.
> > >> >
> > >> > ==== Detecting re-established trusts ====
> > >> > In cases the `sdap_kinit_send()` request fails, the sdap code would
> return a
> > >> > special error code instead of blindly returning `EIO` as it does at
> the
> > >> > moment. When the `ipa_get_ad_acct` request receives this error code, it
> would
> > >> > re-run the subdomain request in order to check if the trust
> relationship still
> > >> > exists and in order to re-fetch the keytab again. In case the trust is
> still
> > >> > established, we would re-set the inactive domain status and re-run the
> account
> > >> > request. In case the trust went away, the subdomain code should remove
> the
> > >> > trusted domain (however, this must be tested). If the account request
> returns
> > >> > any error after reinitializing the ketab, the subdomain would be
> marked
> > >> > inactive in order to avoid fetching the keytab again and again. In
> order to be
> > >> > able to run the subdomain request separately from the subdomain back
> end
> > >> > handler, the subdomain code must be wrapped into a subdomain request as
> the
> > >> > code currently assumes it's being called from the subdomain backend
> handler
> > >> > only.
> > >> >
> > >> > === Future work ===
> > >> > Handling failover and offline status on per-domain basis instead of
> > >> > per-backend basis should be done in the next release.
> > >> >
> > >> > === Configuration changes ===
> > >> > none
> > >> >
> > >> > === How To Test ===
> > >> > Establish a one-way trust relationship with an AD domain. Make sure
> both
> > >> > IPA and AD users are resolvable. It's prudent to test combinations
> of
> > >> > one-way and two-way trusts with different forests. Make sure removing
> a
> > >> > trust relationship removes the keytab from the filesystem. Make sure
> that
> > >> > SSSD handles re-establishing a trust relationship.
> > >> >
> > >> > === Authors ===
> > >> > * Jakub Hrozek <jhrozek(a)redhat.com&gt;
> > >>
> > >> Good job.
> > >
> > >Thank you for taking the time to review the design.
> > >
> > >I made some changes to the design page, here is the diff:
> >
> >https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts?action=diff&version=9&old_version=8
> > >
> > >I'll do one more change after we agree on how should we protect against
> > >loops in fetching keytabs (flag vs. check keytab contents).
> > I think the design is sensible.
>
> Ack.
>
> Simo.
>  
-- 
/ Alexander Bokovoy

[Jakub Hrozek]

On Mon, Apr 27, 2015 at 06:41:42PM +0300, Alexander Bokovoy wrote:
>  On Mon, 27 Apr 2015, Simo Sorce wrote:
>  >On Mon, 2015-04-27 at 15:04 +0300, Alexander Bokovoy wrote:
>  >>On Mon, 27 Apr 2015, Jakub Hrozek wrote:
>  >>>On Sun, Apr 26, 2015 at 06:17:21PM -0400, Simo Sorce wrote:
>  >>>> Very nice writeup!
>  >>>> A few comments..
>  >>>>
>  >>>> On Sun, 2015-04-26 at 21:22 +0200, Jakub Hrozek wrote:
>  >>>> > Hi,
>  >>>> >
>  >>>> > I prepared design page for the one-way trusts feature:
>  >>>> >     https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts
>  >>>> >
>  >>>> > The text is copied below as well for your convenience. I have a
> couple
>  >>>> > of questions to discuss:
>  >>>> >     - Is it worth it to put the keytabs into a new subdirectory so
> that
>  >>>> >       a special SELinux label can be applied? I think yes, since
> then
>  >>>> >       we'd have additional layer of protection of the trust
> secrets
>  >>>>
>  >>>> It is not clear to me what would be the advantage, who are you trying
> to
>  >>>> protect against ?
>  >>>> Adding different labels makes sense only if sssd_be has a different
>  >>>> label from the other sssd processes, but then you can simply have
>  >>>> selinux policies that label accordingly to the parent process in the
>  >>>> same directory, no?
>  >>>
>  >>>Yes, I tried to future-proof the directory structure against different
>  >>>sssd processes that need access to the database (responders) but not the
>  >>>keytabs, because I suspect we won't think about the directory structure
>  >>>again once we set it up :-). And a single separate directory just seemed
>  >>>easier to label with a new context than several files.
>  >>>
>  >>>In the end, I don't care that much as long as the keytabs have different
>  >>>contexts that the responders can't access.
>  >>Given that we want to have one keytab per forest trust and naming
>  >>keytabs after forest trust name, we can live with a single directory. It
>  >>would be good, indeed, to have it separated from the rest of sssd.
>  >>
>  >>
>  >>>>
>  >>>> >     - Is there any reasonable fallback beyond re-fetching the
> keytab we
>  >>>> >       can do? I can't think of any..
>  >>>>
>  >>>> Be careful with refetches and so we do not want to end up in infinite
>  >>>> loops, where the keytab can be fetched but still operation fails for
>  >>>> whatever reason.
>  >>>
>  >>>Sure, I will add a more explicit note once we agree what the mechanism
>  >>>would be.
>  >>Another thing to remember is a potential need to limit enctypes you'd be
>  >>requesting because camellia ciphers are not know to AD and might cause
>  >>issues at some point.
>  >
>  >SSSD should not be in the business of *creating* keytabs, it should only
>  >be allowed to retrieve a precreated key, so SSSD shouldn't care about
>  >enctypes, it will get only those that the FreeIPA code stored in the key
>  >in LDAP.
>  So this means ipasam needs to limit enctypes when asking for the keys.
>  
>  
>  >>>> One thing that should be checked is if the new keytab and old keytab
>  >>>> have the same keys in them, and, if so just stop re-fetching and
> raising
>  >>>> a fatal error (at list for a reasonable period of time).
>  >>>
>  >>>I thought about a flag in the ipa_ad_server_ctx structure, because after
>  >>>the trust is deleted and set up again, the kvno is still 1. When you say
>  >>>'keys', are you proposing to compare the secret key?
> (krb5_keytab_entry.key)
>  >>Yes, don't use KVNO value to compare, verify actual keys.
>  >
>  >It would be nice to avoid the need to link directly to krb5 calls, but
>  >it is probably safer to check by key value. 
About linking to the krb5 calls, I don't think there would be the same
concerns as we had with krb5_child for instance. Because this is a
private sssd keytab, we can make it accessible to the sssd user and all
krb5 calls can run as this user.

>  Note that we are dealing with a key material from a separate realm
> and
>  we create the key in IPA LDAP by means of ipasam which results in the
>  fact there is no way to co-relate KVNO between the key in IPA LDAP and in AD
>  LDAP other than by authenticating or comparing key value to the local
>  copy.
>  
>  On the other hand, when trust is re-established, we actually remove it
>  completely and re-create the objects. This means for new trust KVNO will
>  always be 1 and will only increase if there was a TDO password change
>  operation initiated by AD side (which currently doesn't work, we have
>  this part broken). So I think we probably need to remember a timestamp
>  of when trust object -- not the Kerberos principal but the trust object
>  that ipasam maintains for Samba -- was created/modified and compare that
>  with the time we requested keytab before as a short circuit. As we run
>  on the same server as LDAP server, that time should be comparable. 
Is this data available in krbLastPwdChange ? It's certainly doable to
store that time in sysdb. But since the result of the operation is the
keytab, it seems more natural to me to compare keytab contents and not a
separate (even if logically related) object.

>  
>  >>>> >     - Should the IPA tools signal SSSD either via UNIX signals or
> via
>  >>>> >       the DBus API once a trust has been added or removed to
> minimize
>  >>>> >       the race condition and improve user experience? Currently I
> hope
>  >>>> >       not, the proposed mechanism should be enough but testing is
> still
>  >>>> >       required.
>  >>>> >
>  >>>> > When the design page is agreed on, I'll turn the three
> subsections of
>  >>>> > the implementation phase into smaller tickets and close the
> tracker.
>  >>>> >
>  >>>> > = Feature Name =
>  >>>> >
>  >>>> > Related ticket(s):
>  >>>> >  * https://fedorahosted.org/sssd/ticket/2579
>  >>>> >
>  >>>> > === Problem statement ===
>  >>>> > The next IPA release will support one-way trusts. SSSD needs to add
> support
>  >>>> > for this feature in its server mode.
>  >>>> >
>  >>>> > === Use cases ===
>  >>>> > One-way trust to Active Directory where FreeIPA realm trusts Active
> Directory
>  >>>> > forest using cross-forest trust feature of AD but the AD forest
> does not trust
>  >>>> > FreeIPA realm. Users from AD forest can access resources in FreeIPA
> realm.
>  >>>> >
>  >>>> > === Overview of the solution ===
>  >>>> > At a high level, SSSD needs to examine the trust objects whether
> they
>  >>>> > are one-way or two way trusts. For each one-way trust, SSSD needs
> to fetch
>  >>>> > and store the keytab and use the keytab to secure the connection.
> For
>  >>>> > two-way trusts, we can keep using the existing code that reuses the
> IPA
>  >>>> > realm and the system keytab for both IPA and AD connectins. Care
> must be
>  >>>> > taken to remove keytabs of trusts that were removed as well.
>  >>>> >
>  >>>> > Fetching the keytab would be done by calling the `ipa-getkeytab`
> utility
>  >>>> > for every one-way trust. The keytab would only be (re)fetched if
> it's
>  >>>> > missing or if attempting to use the keytab failed. On the IPA
> server,
>  >>>> > we must make sure that the IPA server identity is allowed to read
> the keytab.
>  >>>> >
>  >>>> > Because handling multiple keytabs increases the risk of failing
> connections in
>  >>>> > case the trust wasn't setup correctly, we need to modify the
> failover code to
>  >>>> > not set the whole back end offline in case connecting to a
> subdomain AD server
>  >>>> > fails. Instead, the subdomain will be marked as inactive for a
> short period of
>  >>>> > time, during which it would act as offline. The proper way of
> solving this
>  >>>> > problem would be to rework the failover module so that it can act
> per domain,
>  >>>> > not only per back end, however, that change is out of scope for
> this release.
>  >>>> >
>  >>>> > === Implementation details ===
>  >>>> > This section describes all the required changes in detail.
>  >>>> >
>  >>>> > ==== Reading the subdomains in the IPA subdomain handler ====
>  >>>> > The IPA subdomain handler will include the attribute
> `ipaNTTrustDirection`
>  >>>> > when reading the trust objects. Currently this attribute is not
> readable
>  >>>> > by the host principal, so the IPA ACIs must be relaxed (ipa
> ticket?). If
>  >>>> > the trust direction is set to an OR of
> `lsa.LSA_TRUST_DIRECTION_OUTBOUND`
>  >>>> > and `lsa.LSA_TRUST_DIRECTION_INBOUND`, then it's a two-way
> trust and we'll
>  >>>> > just use the existing code that re-uses the IPA keytab for the AD
> trusted
>  >>>> > domain as well. If the attribute is only
> `lsa.LSA_TRUST_DIRECTION_OUTBOUND`,
>  >>>> > we handle the trust as a one-way trust. The trust type can be
> stored in
>  >>>> > `ipa_ad_server_ctx`.
>  >>>>
>  >>>> What do you do if you have only lsa.LSA_TRUST_DIRECTION_INBOUND ?
>  >>>
>  >>>Log this object loudly and ignore. Is there another reasonable thing to
>  >>>do other than skip such trust relationship ? (Yes, skipping the object
>  >>>should be spelled out explicitly in the design page)
>  >>Yes, nothing else you can do. I'm not sure at which point we'll get to
>  >>INBOUND-only case.
>  >
>  >We should probably just ignore w/o even screaming too loud.
>  Yep. 
OK, fine. 

>  
>  >
>  >>>> > Each `sss_domain_info` structure will be created as `inactive` in
> the
>  >>>> > subdomain code. After enumerating the trusted domains, the
> subdomain
>  >>>> > handler will check if a keytab already exists for every one-way
> trusted
>  >>>> > domain. If yes, the domain is ready to use and can be enabled. If
> there is
>  >>>> > no keytab, the subdomain handler will fork out a call to
> `ipa-getkeytab`,
>  >>>> > fetch the keytab and store it under `/var/lib/sss/db`. The
> `ipa-getkeytab`
>  >>>> > call will be done using Kerberos credentials the host has. IPA ACIs
> must
>  >>>> > be modified accordingly to allow the IPA server principals to fetch
> the
>  >>>> > trust keytabs, but noone else.
>  >>>> >
>  >>>> > The directory `/var/lib/sss/db` is already only accessible to the
> sssd
>  >>>> > user only. As an additional security measure, we might want to
> store the
>  >>>> > keytabs under a new directory so that the keytabs also receive a
> SELinux
>  >>>> > context stricter than the default `sssd_var_lib_t`. If fetching the
> keytab
>  >>>> > succeeds, the domain would be enabled.  The SELinux policy must
> also be
>  >>>> > adjusted to allow calling `ipa-getkeytab` by the `sssd_be`
> process.
>  >>>>
>  >>>> Getting a keytab is an LDAP extended operation, do you want to fork off
>  >>>> ipa-getkeytab to avoid coding the LDAP operatin in SSSD ? Or for some
>  >>>> other reason ?
>  >>>
>  >>>No, I just wanted to use code that is already there. Retrieving the
>  >>>keytab should be rare enough to justify the overhead of forking an
>  >>>executable and would let us to keep up with any changes IPA might do.
>  >>>
>  >>>Or let me turn that question around -- do you see any advantage of
>  >>>searching the keytab directly from SSSD?
>  >>I can see a downside in the fact that you'd need to handle ASN.1
>  >>encoding/decoding to fetch the keytab and if we ever change its
>  >>structure, SSSD would be another point of failure. Calling out to
>  >>ipa-getkeytab is giving us assurance to keep things working.
>  >
>  >We cannot change the format, if we do we'll have to indtroduce a new
>  >version and support the old one. So this is not a problem.
>  >
>  >The advantage of doing this directly is if we decide (for whatever
>  >reason) to store the key material directly in the ldb cache instead of a
>  >file. If we did that then we could use a MEMORY keytab for the various
>  >operations by fetching it out of the cache and it would not ever hit the
>  >filesystem. Hoewever we may not want to have that key material in LDB at
>  >all, it is just a possibility.
>  Right, it is currently easier to call out and leave ipa-getkeytab to do
>  the job because we are practically running on IPA master in this more. 
I agree.

>  
>  
>  >>>> > If any trust relationships were removed, the corresponding keytabs
> must
>  >>>> > be removed from the disk as well.
>  >>>>
>  >>>> How do you do this ?
>  >>>> Just regularly enumerate the keytab files on disk and remove the ones
>  >>>> that have no object in the cache ?
>  >>>
>  >>>Yes, after the subdomain handler finishes. We alrady have a return boolean
>  >>>'refresh_has_changes' and re-initialize domain-realm mapping for
>  >>>example.
>  >>>
>  >>>> One advantage of using a separate subdir for keytabs, as you propose,
>  >>>> would be to wipe it clean every time sssd restart (at stop and at start
>  >>>> too for good measure), then enumerate the trusts on the server and
>  >>>> re-fetch any keytab.
>  >>>
>  >>>I thought about this, but it seemed a bit wasteful. But now that I think
>  >>>of it, this would present a 'known control point' to start fresh..so
> it
>  >>>might come handy.
>  >>If you do so, please make it triggered via some explicit mechanism, not
>  >>as a default.
>  >
>  >Why not do it at restart ?
>  Do you want to always re-retrive the keys? I was under assumption we are
>  not going to do that. 
> From my point of view, the pro is the ability for admin to refresh the
trust keytabs by simply restarting the sssd service. The con is the
traffic towards IPA and the overhead of fetching and storing the
keytabs, but I think normally sssd restarts on the servers should be
quite rare -- so I would prefer to implement the fetching on restart at
this point.

(snipped the rest)

[Simo Sorce]

On Mon, 2015-04-27 at 18:41 +0300, Alexander Bokovoy wrote:
>  On Mon, 27 Apr 2015, Simo Sorce wrote:
>  >On Mon, 2015-04-27 at 15:04 +0300, Alexander Bokovoy wrote:
>  >> On Mon, 27 Apr 2015, Jakub Hrozek wrote:
>  >> >On Sun, Apr 26, 2015 at 06:17:21PM -0400, Simo Sorce wrote:
>  >> >> Very nice writeup!
>  >> >> A few comments..
>  >> >>
>  >> >> On Sun, 2015-04-26 at 21:22 +0200, Jakub Hrozek wrote:
>  >> >> > Hi,
>  >> >> >
>  >> >> > I prepared design page for the one-way trusts feature:
>  >> >> >     https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts
>  >> >> >
>  >> >> > The text is copied below as well for your convenience. I have a
> couple
>  >> >> > of questions to discuss:
>  >> >> >     - Is it worth it to put the keytabs into a new subdirectory so
> that
>  >> >> >       a special SELinux label can be applied? I think yes, since
> then
>  >> >> >       we'd have additional layer of protection of the trust
> secrets
>  >> >>
>  >> >> It is not clear to me what would be the advantage, who are you trying
> to
>  >> >> protect against ?
>  >> >> Adding different labels makes sense only if sssd_be has a different
>  >> >> label from the other sssd processes, but then you can simply have
>  >> >> selinux policies that label accordingly to the parent process in the
>  >> >> same directory, no?
>  >> >
>  >> >Yes, I tried to future-proof the directory structure against different
>  >> >sssd processes that need access to the database (responders) but not the
>  >> >keytabs, because I suspect we won't think about the directory structure
>  >> >again once we set it up :-). And a single separate directory just seemed
>  >> >easier to label with a new context than several files.
>  >> >
>  >> >In the end, I don't care that much as long as the keytabs have
> different
>  >> >contexts that the responders can't access.
>  >> Given that we want to have one keytab per forest trust and naming
>  >> keytabs after forest trust name, we can live with a single directory. It
>  >> would be good, indeed, to have it separated from the rest of sssd.
>  >>
>  >>
>  >> >>
>  >> >> >     - Is there any reasonable fallback beyond re-fetching the
> keytab we
>  >> >> >       can do? I can't think of any..
>  >> >>
>  >> >> Be careful with refetches and so we do not want to end up in infinite
>  >> >> loops, where the keytab can be fetched but still operation fails for
>  >> >> whatever reason.
>  >> >
>  >> >Sure, I will add a more explicit note once we agree what the mechanism
>  >> >would be.
>  >> Another thing to remember is a potential need to limit enctypes you'd be
>  >> requesting because camellia ciphers are not know to AD and might cause
>  >> issues at some point.
>  >
>  >SSSD should not be in the business of *creating* keytabs, it should only
>  >be allowed to retrieve a precreated key, so SSSD shouldn't care about
>  >enctypes, it will get only those that the FreeIPA code stored in the key
>  >in LDAP.
>  So this means ipasam needs to limit enctypes when asking for the keys. 
Yes it should only ask for encrypts that the AD server on the other side
understand, but only for good measure. The key used is alwasy determined
by the KDC (AD in this case) so having additional keys in the keytab is
not a problem. A problem would rather be to miss enctypes, as the KDC
could decide to encode a ticket/TGT in one of the missing enctypes then
and we would not be able to decrypt.

>  >> >> One thing that should be checked is if the new
> keytab and old keytab
>  >> >> have the same keys in them, and, if so just stop re-fetching and
> raising
>  >> >> a fatal error (at list for a reasonable period of time).
>  >> >
>  >> >I thought about a flag in the ipa_ad_server_ctx structure, because after
>  >> >the trust is deleted and set up again, the kvno is still 1. When you say
>  >> >'keys', are you proposing to compare the secret key?
> (krb5_keytab_entry.key)
>  >> Yes, don't use KVNO value to compare, verify actual keys.
>  >
>  >It would be nice to avoid the need to link directly to krb5 calls, but
>  >it is probably safer to check by key value.
>  Note that we are dealing with a key material from a separate realm and
>  we create the key in IPA LDAP by means of ipasam which results in the
>  fact there is no way to co-relate KVNO between the key in IPA LDAP and in AD
>  LDAP other than by authenticating or comparing key value to the local
>  copy. 
We need to be careful with this, IIRC there are some check on KVNO, we
may have to force the KVNO to always be 0 (or 1 ?) to avoid the checks.

>  On the other hand, when trust is re-established, we actually remove
> it
>  completely and re-create the objects. This means for new trust KVNO will
>  always be 1 and will only increase if there was a TDO password change
>  operation initiated by AD side (which currently doesn't work, we have
>  this part broken). So I think we probably need to remember a timestamp
>  of when trust object -- not the Kerberos principal but the trust object
>  that ipasam maintains for Samba -- was created/modified and compare that
>  with the time we requested keytab before as a short circuit. As we run
>  on the same server as LDAP server, that time should be comparable.
>  
>  >> >> >     - Should the IPA tools signal SSSD either via UNIX signals or
> via
>  >> >> >       the DBus API once a trust has been added or removed to
> minimize
>  >> >> >       the race condition and improve user experience? Currently I
> hope
>  >> >> >       not, the proposed mechanism should be enough but testing is
> still
>  >> >> >       required.
>  >> >> >
>  >> >> > When the design page is agreed on, I'll turn the three
> subsections of
>  >> >> > the implementation phase into smaller tickets and close the
> tracker.
>  >> >> >
>  >> >> > = Feature Name =
>  >> >> >
>  >> >> > Related ticket(s):
>  >> >> >  * https://fedorahosted.org/sssd/ticket/2579
>  >> >> >
>  >> >> > === Problem statement ===
>  >> >> > The next IPA release will support one-way trusts. SSSD needs to
> add support
>  >> >> > for this feature in its server mode.
>  >> >> >
>  >> >> > === Use cases ===
>  >> >> > One-way trust to Active Directory where FreeIPA realm trusts
> Active Directory
>  >> >> > forest using cross-forest trust feature of AD but the AD forest
> does not trust
>  >> >> > FreeIPA realm. Users from AD forest can access resources in
> FreeIPA realm.
>  >> >> >
>  >> >> > === Overview of the solution ===
>  >> >> > At a high level, SSSD needs to examine the trust objects whether
> they
>  >> >> > are one-way or two way trusts. For each one-way trust, SSSD needs
> to fetch
>  >> >> > and store the keytab and use the keytab to secure the connection.
> For
>  >> >> > two-way trusts, we can keep using the existing code that reuses
> the IPA
>  >> >> > realm and the system keytab for both IPA and AD connectins. Care
> must be
>  >> >> > taken to remove keytabs of trusts that were removed as well.
>  >> >> >
>  >> >> > Fetching the keytab would be done by calling the `ipa-getkeytab`
> utility
>  >> >> > for every one-way trust. The keytab would only be (re)fetched if
> it's
>  >> >> > missing or if attempting to use the keytab failed. On the IPA
> server,
>  >> >> > we must make sure that the IPA server identity is allowed to read
> the keytab.
>  >> >> >
>  >> >> > Because handling multiple keytabs increases the risk of failing
> connections in
>  >> >> > case the trust wasn't setup correctly, we need to modify the
> failover code to
>  >> >> > not set the whole back end offline in case connecting to a
> subdomain AD server
>  >> >> > fails. Instead, the subdomain will be marked as inactive for a
> short period of
>  >> >> > time, during which it would act as offline. The proper way of
> solving this
>  >> >> > problem would be to rework the failover module so that it can act
> per domain,
>  >> >> > not only per back end, however, that change is out of scope for
> this release.
>  >> >> >
>  >> >> > === Implementation details ===
>  >> >> > This section describes all the required changes in detail.
>  >> >> >
>  >> >> > ==== Reading the subdomains in the IPA subdomain handler ====
>  >> >> > The IPA subdomain handler will include the attribute
> `ipaNTTrustDirection`
>  >> >> > when reading the trust objects. Currently this attribute is not
> readable
>  >> >> > by the host principal, so the IPA ACIs must be relaxed (ipa
> ticket?). If
>  >> >> > the trust direction is set to an OR of
> `lsa.LSA_TRUST_DIRECTION_OUTBOUND`
>  >> >> > and `lsa.LSA_TRUST_DIRECTION_INBOUND`, then it's a two-way
> trust and we'll
>  >> >> > just use the existing code that re-uses the IPA keytab for the AD
> trusted
>  >> >> > domain as well. If the attribute is only
> `lsa.LSA_TRUST_DIRECTION_OUTBOUND`,
>  >> >> > we handle the trust as a one-way trust. The trust type can be
> stored in
>  >> >> > `ipa_ad_server_ctx`.
>  >> >>
>  >> >> What do you do if you have only lsa.LSA_TRUST_DIRECTION_INBOUND ?
>  >> >
>  >> >Log this object loudly and ignore. Is there another reasonable thing to
>  >> >do other than skip such trust relationship ? (Yes, skipping the object
>  >> >should be spelled out explicitly in the design page)
>  >> Yes, nothing else you can do. I'm not sure at which point we'll get to
>  >> INBOUND-only case.
>  >
>  >We should probably just ignore w/o even screaming too loud.
>  Yep.
>  
>  >
>  >> >> > Each `sss_domain_info` structure will be created as `inactive` in
> the
>  >> >> > subdomain code. After enumerating the trusted domains, the
> subdomain
>  >> >> > handler will check if a keytab already exists for every one-way
> trusted
>  >> >> > domain. If yes, the domain is ready to use and can be enabled. If
> there is
>  >> >> > no keytab, the subdomain handler will fork out a call to
> `ipa-getkeytab`,
>  >> >> > fetch the keytab and store it under `/var/lib/sss/db`. The
> `ipa-getkeytab`
>  >> >> > call will be done using Kerberos credentials the host has. IPA
> ACIs must
>  >> >> > be modified accordingly to allow the IPA server principals to
> fetch the
>  >> >> > trust keytabs, but noone else.
>  >> >> >
>  >> >> > The directory `/var/lib/sss/db` is already only accessible to the
> sssd
>  >> >> > user only. As an additional security measure, we might want to
> store the
>  >> >> > keytabs under a new directory so that the keytabs also receive a
> SELinux
>  >> >> > context stricter than the default `sssd_var_lib_t`. If fetching
> the keytab
>  >> >> > succeeds, the domain would be enabled.  The SELinux policy must
> also be
>  >> >> > adjusted to allow calling `ipa-getkeytab` by the `sssd_be`
> process.
>  >> >>
>  >> >> Getting a keytab is an LDAP extended operation, do you want to fork
> off
>  >> >> ipa-getkeytab to avoid coding the LDAP operatin in SSSD ? Or for some
>  >> >> other reason ?
>  >> >
>  >> >No, I just wanted to use code that is already there. Retrieving the
>  >> >keytab should be rare enough to justify the overhead of forking an
>  >> >executable and would let us to keep up with any changes IPA might do.
>  >> >
>  >> >Or let me turn that question around -- do you see any advantage of
>  >> >searching the keytab directly from SSSD?
>  >> I can see a downside in the fact that you'd need to handle ASN.1
>  >> encoding/decoding to fetch the keytab and if we ever change its
>  >> structure, SSSD would be another point of failure. Calling out to
>  >> ipa-getkeytab is giving us assurance to keep things working.
>  >
>  >We cannot change the format, if we do we'll have to indtroduce a new
>  >version and support the old one. So this is not a problem.
>  >
>  >The advantage of doing this directly is if we decide (for whatever
>  >reason) to store the key material directly in the ldb cache instead of a
>  >file. If we did that then we could use a MEMORY keytab for the various
>  >operations by fetching it out of the cache and it would not ever hit the
>  >filesystem. Hoewever we may not want to have that key material in LDB at
>  >all, it is just a possibility.
>  Right, it is currently easier to call out and leave ipa-getkeytab to do
>  the job because we are practically running on IPA master in this more. 
Fine, I just wanted to know the reasons.

>  >> >> > If any trust relationships were removed, the
> corresponding keytabs must
>  >> >> > be removed from the disk as well.
>  >> >>
>  >> >> How do you do this ?
>  >> >> Just regularly enumerate the keytab files on disk and remove the ones
>  >> >> that have no object in the cache ?
>  >> >
>  >> >Yes, after the subdomain handler finishes. We alrady have a return boolean
>  >> >'refresh_has_changes' and re-initialize domain-realm mapping for
>  >> >example.
>  >> >
>  >> >> One advantage of using a separate subdir for keytabs, as you propose,
>  >> >> would be to wipe it clean every time sssd restart (at stop and at
> start
>  >> >> too for good measure), then enumerate the trusts on the server and
>  >> >> re-fetch any keytab.
>  >> >
>  >> >I thought about this, but it seemed a bit wasteful. But now that I think
>  >> >of it, this would present a 'known control point' to start fresh..so
> it
>  >> >might come handy.
>  >> If you do so, please make it triggered via some explicit mechanism, not
>  >> as a default.
>  >
>  >Why not do it at restart ?
>  Do you want to always re-retrive the keys? I was under assumption we are
>  not going to do that. 
Well we can have an option, or always do that, restarts are rare, esp on
a server, so I thought it may be a simple way to start from a known
clean slate.

>  >> >> > ==== Changes to the AD id_ctx instantiation
> ====
>  >> >> > With two-way trust, we can keep using the default IPA principal
> and keytab.
>  >> >> >
>  >> >> > With one-way trust, the keytab retrieved from the IPA server must
> be
>  >> >> > used. Also, the principal must be passed into the
> `ad_create_default_options`
>  >> >> > function. The custom values must be set before we proceed to
> instantiate
>  >> >> > LDAP provider options. The only AD provider option we need to set
> is
>  >> >> > AD_KRB5_REALM.
>  >> >> >
>  >> >> > In the LDAP provider, we must take care that the following sdap
> options
>  >> >> > are set correctly:
>  >> >> >     * SDAP_SASL_AUTHID - must be set to the NetBIOS name of the
> IPA domain. (A
>  >> >> >       domain `TRUST.COM` would set this value to `TRUST$`).
>  >> >>
>  >> >> The NetBIOS domain can be completely unrelated to the DNS domain, so
>  >> >> make sure you do not just try to truncate the realm but actually fetch
>  >> >> the info from IPA/AD.
>  >> >
>  >> >Sure, sorry, that was meant as an example only. I added a more explicit
>  >> >sentence.
>  >> >
>  >> >>
>  >> >> >     * SDAP_SASL_REALM - must be set to the AD realm
>  >> >> >     * SDAP_KRB5_KEYTAB - must be set to the per-domain keytab
> retrieved from IPA
>  >> >> >
>  >> >> > The AD provider eventually calls `sdap_set_sasl_options()` from
> the
>  >> >> > LDAP provider, we need to make sure this function receives the
> correct
>  >> >> > values. During experimentation we were able to show that using
> multiple
>  >> >> > different SASL users and different realms doesn't cause any
> problems in
>  >> >> > SASL or LDAP libraries.
>  >> >> >
>  >> >> > The only place that will keep using the IPA realm is the failover
>  >> >> > instantiation. We need to keep using this hack until failover is
> per-backend.
>  >> >> >
>  >> >> > ==== Subdomain offline status changes ====
>  >> >> > At the moment, the whole back end can be either online or offline
> and
>  >> >> > the status applies to both the main domain and the subdomains
> alike. As
>  >> >> > an effect, a failure to connect to a subdomain server would also
> make the
>  >> >> > main domain operate offline. In many subdomain setups, it's
> actually more
>  >> >> > convenient not to, because the subdomain server might be on a
> different
>  >> >> > network segment, behind a different firewall etc. Instead, the
> domain
>  >> >> > should only be made inactive.
>  >> >>
>  >> >> Does this cover accurately the Forest vs AD subdomain cases ?
>  >> >> We will have a keytab only for the AD root domain (the forest) and we
>  >> >> need to use those credentials for any other domain in the AD forest.
>  >> >
>  >> >Well, both subdomains would be made inactive, then. The point is to
>  >> >avoid marking the IPA domain as offline if the trusts break.
>  >> Yes, if you don't have keytab with keys to TDO object, you cannot
>  >> authenticate against any of the forest's AD DCs, in any domain of the
>  >> forest, so all of them would be offline in this case. Our current
>  >> problem is that we offline IPA domain here instead of the subdomains.
>  >>
>  >>
>  >> >btw this would be also useful for AD-AD trusts because what I learned is
>  >> >that AD subdomains tend to be isolated from one another on the network
>  >> >level as well and then failure to reach other domains is currently fatal
>  >> >which brings the whole sssd_be process offline.
>  >> >
>  >> >>
>  >> >> > The `sss_domain_info` structure would convert the 'bool
> disabled' parameter
>  >> >> > into an `enum sss_domain_state`. The supported values would be:
>  >> >> >     * ''disabled'' - the domain should not be used
> by either responder or
>  >> >> >       provider. It was removed or disabled on the server.
>  >> >> >     * ''active'' - the domain can be used by a
> responder and the data provider
>  >> >> >       would forward its request to the backend
>  >> >> >     * ''inactive'' - the domain can be used by a
> responder, but the data
>  >> >> >       provider would just shortcut as if the domain was offline.
> For now, this
>  >> >> >       option will be used by subdomains only.
>  >> >> >
>  >> >> > The implementation would include renaming the existing
> `be_mark_offline()`
>  >> >> > function to be called `be_mark_dom_offline()` and modifying its
> behaviour. The
>  >> >> > existing code that sets the offline status and runs the offline
> callbacks
>  >> >> > would be called for parent domains only. For subdomains, we would
> mark the
>  >> >> > subdomain as inactive and schedule a tevent request that would
> unconditionally
>  >> >> > reset inactive domain to active. The request would be scheduled
> after
>  >> >> > `offline_timeout` seconds to be consistent with main domains from
> user's
>  >> >> > perspective. Likewise, the `be_reset_offline()` function will be
> extended to
>  >> >> > reset inactive domains to active as well as the SIGUSR1 and
> SIGUSR2 signal
>  >> >> > handlers. Finally, all calls to the `be_is_offline()` function
> should be
>  >> >> > inspected and the invocations that are per-domain should be
> converted to a new
>  >> >> > function `be_dom_is_offline()` that would check the offline status
> for parent
>  >> >> > domains and the offline state for subdomains. We should also make
> sure the
>  >> >> > backend offline status structure is opaque as currently its
> internals are
>  >> >> > readable by external users as well. Making the offline status
> opaque would
>  >> >> > make it safer to perform modifications to the offline code.
>  >> >> >
>  >> >> > In both offline and inactive cases, the ID handlers would reply
> with
>  >> >> > `DP_ERR_OFFLINE`. The crucial difference between offline and
> inactive at this
>  >> >> > point would be that inactive domains are re-activated
> undonditionally. When we
>  >> >> > modify the failover code to handle domains separately, we'll
> be able to
>  >> >> > leverage per-domain online checks or online/offline callbacks as
> well.
>  >> >> >
>  >> >> > ==== Detecting re-established trusts ====
>  >> >> > In cases the `sdap_kinit_send()` request fails, the sdap code
> would return a
>  >> >> > special error code instead of blindly returning `EIO` as it does
> at the
>  >> >> > moment. When the `ipa_get_ad_acct` request receives this error
> code, it would
>  >> >> > re-run the subdomain request in order to check if the trust
> relationship still
>  >> >> > exists and in order to re-fetch the keytab again. In case the
> trust is still
>  >> >> > established, we would re-set the inactive domain status and re-run
> the account
>  >> >> > request. In case the trust went away, the subdomain code should
> remove the
>  >> >> > trusted domain (however, this must be tested). If the account
> request returns
>  >> >> > any error after reinitializing the ketab, the subdomain would be
> marked
>  >> >> > inactive in order to avoid fetching the keytab again and again. In
> order to be
>  >> >> > able to run the subdomain request separately from the subdomain
> back end
>  >> >> > handler, the subdomain code must be wrapped into a subdomain
> request as the
>  >> >> > code currently assumes it's being called from the subdomain
> backend handler
>  >> >> > only.
>  >> >> >
>  >> >> > === Future work ===
>  >> >> > Handling failover and offline status on per-domain basis instead
> of
>  >> >> > per-backend basis should be done in the next release.
>  >> >> >
>  >> >> > === Configuration changes ===
>  >> >> > none
>  >> >> >
>  >> >> > === How To Test ===
>  >> >> > Establish a one-way trust relationship with an AD domain. Make
> sure both
>  >> >> > IPA and AD users are resolvable. It's prudent to test
> combinations of
>  >> >> > one-way and two-way trusts with different forests. Make sure
> removing a
>  >> >> > trust relationship removes the keytab from the filesystem. Make
> sure that
>  >> >> > SSSD handles re-establishing a trust relationship.
>  >> >> >
>  >> >> > === Authors ===
>  >> >> > * Jakub Hrozek <jhrozek(a)redhat.com&gt;
>  >> >>
>  >> >> Good job.
>  >> >
>  >> >Thank you for taking the time to review the design.
>  >> >
>  >> >I made some changes to the design page, here is the diff:
>  >>
> >https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts?action=diff&version=9&old_version=8
>  >> >
>  >> >I'll do one more change after we agree on how should we protect against
>  >> >loops in fetching keytabs (flag vs. check keytab contents).
>  >> I think the design is sensible.
>  >
>  >Ack.
>  >
>  >Simo.
>  >
>

[Jakub Hrozek]

On Mon, Apr 27, 2015 at 10:32:03AM +0200, Jakub Hrozek wrote:
>  Sure, I will add a more explicit note once we agree what the
> mechanism
>  would be. 
I modified the design page to agree with this discussion:
   
https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts?action=diff&am...

The changes include:
    - noted that we fetch all enctypes from IPA and that IPA is
      responsible for filtering/requesting the right keytabs
    - note we would prune and fetch the keytabs on restart. If we see
      during development that this is taking too much time, we can back
      off.
    - there is a note that inbound trusts are ignored
    - there is a note why we're calling ipa-getkeytab explicitly and why
      we might consider moving to calling the extop ourselves in the
      future
    - keytab comparison is spelled out more explicitly (keys are
      compared) and there is an explicit note that krb5 calls don't
      hurt because the keytab is owned by the sssd user already.

I'll file the per-task tickets now.

[Alexander Bokovoy]

On Tue, 28 Apr 2015, Jakub Hrozek wrote:
> On Mon, Apr 27, 2015 at 10:32:03AM +0200, Jakub Hrozek wrote:
> > Sure, I will add a more explicit note once we agree what the mechanism
> > would be.
>
> I modified the design page to agree with this discussion:
>    
> https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts?action=diff&am...
>
> The changes include:
>     - noted that we fetch all enctypes from IPA and that IPA is
>       responsible for filtering/requesting the right keytabs
>     - note we would prune and fetch the keytabs on restart. If we see
>       during development that this is taking too much time, we can back
>       off.
>     - there is a note that inbound trusts are ignored
>     - there is a note why we're calling ipa-getkeytab explicitly and why
>       we might consider moving to calling the extop ourselves in the
>       future
>     - keytab comparison is spelled out more explicitly (keys are
>       compared) and there is an explicit note that krb5 calls don't
>       hurt because the keytab is owned by the sssd user already.
>
> I'll file the per-task tickets now. 
ACK. Do you need FreeIPA tickets too? Just
file them as well.
-- 
/ Alexander Bokovoy

[Jakub Hrozek]

On Tue, Apr 28, 2015 at 01:34:56PM +0300, Alexander Bokovoy wrote:
>  On Tue, 28 Apr 2015, Jakub Hrozek wrote:
>  >On Mon, Apr 27, 2015 at 10:32:03AM +0200, Jakub Hrozek wrote:
>  >>Sure, I will add a more explicit note once we agree what the mechanism
>  >>would be.
>  >
>  >I modified the design page to agree with this discussion:
>  >  
> https://fedorahosted.org/sssd/wiki/DesignDocs/OneWayTrusts?action=diff&am...
>  >
>  >The changes include:
>  >   - noted that we fetch all enctypes from IPA and that IPA is
>  >     responsible for filtering/requesting the right keytabs
>  >   - note we would prune and fetch the keytabs on restart. If we see
>  >     during development that this is taking too much time, we can back
>  >     off.
>  >   - there is a note that inbound trusts are ignored
>  >   - there is a note why we're calling ipa-getkeytab explicitly and why
>  >     we might consider moving to calling the extop ourselves in the
>  >     future
>  >   - keytab comparison is spelled out more explicitly (keys are
>  >     compared) and there is an explicit note that krb5 calls don't
>  >     hurt because the keytab is owned by the sssd user already.
>  >
>  >I'll file the per-task tickets now.
>  ACK. Do you need FreeIPA tickets too? Just file them as well. 
OK, I filed these SSSD tickets:
    https://fedorahosted.org/sssd/ticket/2636
    https://fedorahosted.org/sssd/ticket/2637
    https://fedorahosted.org/sssd/ticket/2638
    https://fedorahosted.org/sssd/ticket/2639

And two IPA tickets:
    https://fedorahosted.org/freeipa/ticket/5004
    https://fedorahosted.org/freeipa/ticket/5005