Hey,
So I am trying to implement TOTP+password for SSH on a server. In the past its been as simple as using google authenticatior but seeing as how we have a shiny FreeIPA server...
Created a user, then gave them a TOTP token (synched and tested that it works by logging into the web ui). But I'm stuck at the correct way to implement this on the SSH server. Found the earlier thread[1] and got some pointers. sshd config:
ChallengeResponseAuthentication yes AuthenticationMethods keyboard-interactive
If I do not define password/otp for the host via the IPA web interface, login works fine with password. If I set it to password/otp only it fails.
Looking at journalctl -xeu ssh.service there clearly is some issue.
pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): received for user kjell: 7 (Authentication failure) error: PAM: Authentication failure for kjell from 192.168.31.102 pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): received for user kjell: 4 (System error) error: PAM: Authentication failure for kjell from 192.168.31.102 Postponed keyboard-interactive for kjell from 192.168.31.102 port 38832 ssh2 [preauth] pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): received for user kjell: 4 (System error) error: PAM: Authentication failure for kjell from 192.168.31.102 Failed keyboard-interactive/pam for kjell from 192.168.31.102 port 38832 ssh2 Connection closed by authenticating user kjell 192.168.31.102 port 38832 [preauth]
Tried giving my password, and my password+otp (without the '+'). But nothing works.
Anyone got any pointers or see any obvious mistakes ?
1: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahost...
On pe, 23 joulu 2022, Kjell Cornelius Nicolaysen via FreeIPA-users wrote:
Hey,
So I am trying to implement TOTP+password for SSH on a server. In the past its been as simple as using google authenticatior but seeing as how we have a shiny FreeIPA server...
Created a user, then gave them a TOTP token (synched and tested that it works by logging into the web ui). But I'm stuck at the correct way to implement this on the SSH server. Found the earlier thread[1] and got some pointers. sshd config:
ChallengeResponseAuthentication yes AuthenticationMethods keyboard-interactive
If I do not define password/otp for the host via the IPA web interface, login works fine with password. If I set it to password/otp only it fails.
Looking at journalctl -xeu ssh.service there clearly is some issue.
pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): received for user kjell: 7 (Authentication failure) error: PAM: Authentication failure for kjell from 192.168.31.102 pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): received for user kjell: 4 (System error) error: PAM: Authentication failure for kjell from 192.168.31.102 Postponed keyboard-interactive for kjell from 192.168.31.102 port 38832 ssh2 [preauth] pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): received for user kjell: 4 (System error) error: PAM: Authentication failure for kjell from 192.168.31.102 Failed keyboard-interactive/pam for kjell from 192.168.31.102 port 38832 ssh2 Connection closed by authenticating user kjell 192.168.31.102 port 38832 [preauth]
Tried giving my password, and my password+otp (without the '+'). But nothing works.
Anyone got any pointers or see any obvious mistakes ?
You get system error from pam_sss. You need to enable debug logging in SSSD and collect logs. Please see https://sssd.io/troubleshooting/basics.html for more details.
So here is a twist on getting SSH/OTP to work. If I use the web UI and:
1) set user authentication types to "Two factor authentication (password + OTP), then set the host I am trying to login to to "password+OTP", it works.
2) if the user has no authentication types hooked, or has Hardened Password or Password in addition to the 2FA option hooked, login fails.
Had a look at SSSD logs (with debug level 7) but afraid I cannot spot any clear issues save "pre authentication failed" if I have any of the settings mentioned in point 2 above (have tried tracing that but cannot for the life of me find the reason why).
All I am trying to do is require password+otp for the SSH portion. Sudo should only require password, not password and otp... Sorry, but very fresh to FreeIPA so I am certain there is some concept at play here which I am just not seeing.
On 23/12/2022 20:28, Alexander Bokovoy wrote:
On pe, 23 joulu 2022, Kjell Cornelius Nicolaysen via FreeIPA-users wrote:
Hey,
So I am trying to implement TOTP+password for SSH on a server. In the past its been as simple as using google authenticatior but seeing as how we have a shiny FreeIPA server...
Created a user, then gave them a TOTP token (synched and tested that it works by logging into the web ui). But I'm stuck at the correct way to implement this on the SSH server. Found the earlier thread[1] and got some pointers. sshd config:
ChallengeResponseAuthentication yes AuthenticationMethods keyboard-interactive
If I do not define password/otp for the host via the IPA web interface, login works fine with password. If I set it to password/otp only it fails.
Looking at journalctl -xeu ssh.service there clearly is some issue.
pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): received for user kjell: 7 (Authentication failure) error: PAM: Authentication failure for kjell from 192.168.31.102 pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): received for user kjell: 4 (System error) error: PAM: Authentication failure for kjell from 192.168.31.102 Postponed keyboard-interactive for kjell from 192.168.31.102 port 38832 ssh2 [preauth] pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): received for user kjell: 4 (System error) error: PAM: Authentication failure for kjell from 192.168.31.102 Failed keyboard-interactive/pam for kjell from 192.168.31.102 port 38832 ssh2 Connection closed by authenticating user kjell 192.168.31.102 port 38832 [preauth]
Tried giving my password, and my password+otp (without the '+'). But nothing works.
Anyone got any pointers or see any obvious mistakes ?
You get system error from pam_sss. You need to enable debug logging in SSSD and collect logs. Please see https://sssd.io/troubleshooting/basics.html for more details.
On ti, 27 joulu 2022, Kjell Cornelius Nicolaysen wrote:
So here is a twist on getting SSH/OTP to work. If I use the web UI and:
- set user authentication types to "Two factor authentication
(password + OTP), then set the host I am trying to login to to "password+OTP", it works.
- if the user has no authentication types hooked, or has Hardened
Password or Password in addition to the 2FA option hooked, login fails.
This is as designed. If you have forced authentication indicator on the host service to 'otp' then any Kerberos ticket presented to KDC when asking for a ticket to that host/.. service must be obtained with 'otp'.
Please read https://freeipa.readthedocs.io/en/latest/workshop/11-kerberos-ticket-policy...., especially 'Enforcing authentication indicators' section but you need to familiarize yourself with the other parts of the same unit as well.
Had a look at SSSD logs (with debug level 7) but afraid I cannot spot any clear issues save "pre authentication failed" if I have any of the settings mentioned in point 2 above (have tried tracing that but cannot for the life of me find the reason why).
All I am trying to do is require password+otp for the SSH portion. Sudo should only require password, not password and otp... Sorry, but very fresh to FreeIPA so I am certain there is some concept at play here which I am just not seeing.
This is not possible, in general. This is cannot be achieved without maintaining an authentication session trail and that session trail is not flexible enough to allow what you want. It is not a problem of FreeIPA, it is a general complexity of application-specific solutions that cannot really be solved here. We can do that in certain way with Kerberos tickets but even there exists no way to enforce a specific authentication process in case an otherwise valid Kerberos ticket is unfit for a specific purpose.
You either have a ticket granting ticket (TGT) with certain properties because you have obtained it somewhere using a particular pre-authentication method or you don't have it and there is no logic anywhere that would force you to loop back to initial authentication step.
When using Kerberos, the process of authentication to a service really is broken down into several independent authentication and authorization steps: - [a] obtain an initial ticket granting ticket, TGT - [b] use TGT to request a service ticket to the specific Kerberos service - [c] use application-specific protocol to present a service ticket to the application's service to be authenticated - [d] after authentication happened, application needs to authorize the access
The important part here is that there are two independent Kerberos operations where a client side communicates with KDC and KDC might apply some access controls: - request an initial TGT ([a]) - request a service ticket using that initial TGT ([b])
After these steps were done, an application client side does [c] and a application server side does [d]. [d] is where another access control step is typically applied.
An authentication indicator is a mark in the TGT that tells which method was used to authenticate the issuance of the TGT (step [a]). If a service ticket was issued by the KDC, that ticket will contain the same authentication indicator as was in the TGT. Any new indicator cannot be added at any step past [a].
A networking application which uses Kerberos typically does not deal with step [a] directly. Most of them expect a TGT to exist in a credentials cache on the client side. This ticket is used to obtain a service ticket (step [b]) or, if such ticket exists in the credentials cache, it will be reused. For all such applications (SSH is no exception) if Kerberos ticket exists, they don't care about the specifics of its properties other than whether the ticket is valid.
If you are not using Kerberos (GSSAPI) in the first place to login to SSH server, a default SSH server configuration on IPA clients would drive you through the SSH server's PAM stack to authenticate. The PAM stack definition for SSH server typically includes pam_sss.so module. You can think of it as a 'client app' in the description above now.
pam_sss.so module would ask SSSD backend to acquire a Kerberos ticket for you. SSSD would ask for a password+otp if KDC advertises the 'otp' pre-authentication method and Kerberos library on the system supports 'otp' method. pam_sss.so module would pass this request as a text message to the application (in this case, internals of SSH server process) and SSH would use its own protocol to tell to SSH client to show that message to the user.
But if the Kerberos library does not support 'otp' method or KDC does not advertise it, SSSD will happily use the password SSH has provided to pam_sss.so as a part of that communication to obtain a 'normal' TGT, e.g. the one that does not have 'otp' authentication indicator. If user is allowed to use a plain password to authenticate, this request would succeed.
The TGT obtained by SSSD will then be used to ask for a service ticket to host/... principal to validate your ticket (regardless whether it was obtained by SSSD itself or presented from SSH client). This is a step [b]. If you set 'otp' on the host/... principal, KDC will reject any request which provided an evidence of the TGT without 'otp' authentication indicator. There is no way to fall back from here to request a user to obtain a new TGT with other method. At least, I don't know about any application that implements this using Kerberos.
On 23/12/2022 20:28, Alexander Bokovoy wrote:
On pe, 23 joulu 2022, Kjell Cornelius Nicolaysen via FreeIPA-users wrote:
Hey,
So I am trying to implement TOTP+password for SSH on a server. In the past its been as simple as using google authenticatior but seeing as how we have a shiny FreeIPA server...
Created a user, then gave them a TOTP token (synched and tested that it works by logging into the web ui). But I'm stuck at the correct way to implement this on the SSH server. Found the earlier thread[1] and got some pointers. sshd config:
ChallengeResponseAuthentication yes AuthenticationMethods keyboard-interactive
If I do not define password/otp for the host via the IPA web interface, login works fine with password. If I set it to password/otp only it fails.
Looking at journalctl -xeu ssh.service there clearly is some issue.
pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): received for user kjell: 7 (Authentication failure) error: PAM: Authentication failure for kjell from 192.168.31.102 pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): received for user kjell: 4 (System error) error: PAM: Authentication failure for kjell from 192.168.31.102 Postponed keyboard-interactive for kjell from 192.168.31.102 port 38832 ssh2 [preauth] pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.31.102 user=kjell pam_sss(sshd:auth): received for user kjell: 4 (System error) error: PAM: Authentication failure for kjell from 192.168.31.102 Failed keyboard-interactive/pam for kjell from 192.168.31.102 port 38832 ssh2 Connection closed by authenticating user kjell 192.168.31.102 port 38832 [preauth]
Tried giving my password, and my password+otp (without the '+'). But nothing works.
Anyone got any pointers or see any obvious mistakes ?
You get system error from pam_sss. You need to enable debug logging in SSSD and collect logs. Please see https://sssd.io/troubleshooting/basics.html for more details.
-- Mvh, Kjell C. Nicolaysen Bitfrost AS
PGP Public key available on request. Current key (at time of this email) fingerprint: 3F59 7410 AFD5 FC22 F2F1 EEC9 980A 8C9E C126 6716
Kjell,
I have a working solution using Kerberos TGTs to ssh between servers within the realm using OTP. This solution requires that you log into one realm server with password and otp. From there you can ssh to the rest of the realm via kerberos tickets.
I only got to this state by trial and error so some of my statements and assumptions could be wrong. My servers are a mix of Rhel 7.9 and Rhel8.x My IDM servers are Rhel8 latest.
On the ssh server you need to enable GSSAPIAuthentication in sshd_config.
On all the clients you also need to enable GSSAPIAuthentication in ssh_config,
There are some gotchas.
You can't ssh to the ssh server by IP address. GSSAPIAuthentication passes the request to IDM by hostname. So ssh user@192.168.1.101 will fail back to password via pam.sss
On Rhel7 you can only ssh to the FQDN of the ssh server. So you should do: ssh user@myserver.realm.com .
If the Client is Rhel8 you should be able to ssh to the short hostname such as ssh user@myserver .
On Rhel7 clients getting the ssh to work with short hostnames is problematic. You can resolve the problem with the following settings in ssh_config: CanonicalizeHostname always CanonicalDomains myrealm.com CanonicalizeFallbackLocal yes CanonicalizeMaxDots 0 GSSAPITrustDns yes Host *.myrealm.com GSSAPIAuthentication yes GSSAPIDelegateCredentials yes Host ! *.myrealm.com GSSAPIAuthentication no
Your experience with users that don't have an OTP being unable to log into a host that requires OTP is expected.
There are plenty of edge cases around caching, so test before production.
It is also problematic to get a new ticket with kinit if you use OTP. There are steps to get around it, but I am waiting for the solution to be a native Redhat process.
We regularly get this TGT ssh method to fail, when there are too may expired tickets on the ssh server. Once logged into the ssh server by password OtP, you can run kdestroy -A to clear cache and you can again ssh via TGTs.
Obviously ssh and sshd should be configured to failback to username/password+otp with: ChallengeResponseAuthentication yes
freeipa-users@lists.fedorahosted.org
-
Alexander Bokovoy -
Bob Strachan -
Kjell Cornelius Nicolaysen