mod_nss fails apache start missing existing certificate.

[Naor Weissmann]

Hi guys.
I understand it is not a pure FreeIPA question but it is supporting middleware and im out of ideas. 
We have an old ipa-server-selinux-3.0.0 on Centos6. 
after restart i cant start http service. error log in debug mode points me to nss.
"[error] Certificate not found: 'Server-Cert'"
certutil on the database returns:
Server-Cert                                                  CPu,Cu,u
and checks for validity. selinux is in permissive mode. 
if i remove password file i get a password error so mod_nss is going in right direction. 
i dont know what else to look for. maybe you have any ideas? thank you in advance.

[Rob Crittenden]

Naor Weissmann via FreeIPA-users wrote:
> Hi guys.
> I understand it is not a pure FreeIPA question but it is supporting middleware and im out of ideas. 
> We have an old ipa-server-selinux-3.0.0 on Centos6. 
> after restart i cant start http service. error log in debug mode points me to nss.
> "[error] Certificate not found: 'Server-Cert'"
> certutil on the database returns:
> Server-Cert                                                  CPu,Cu,u
> and checks for validity. selinux is in permissive mode. 
> if i remove password file i get a password error so mod_nss is going in right direction. 
> i dont know what else to look for. maybe you have any ideas? thank you in advance.

What is the history with this? Was the certificate recently replaced?
When did it stop working?

rob

[Naor Weissmann]

thank you!
 the cert renewed last year, by itself it is ok and working (it is two years one). However im having issues to add servers to FreeIPA, since it is done via 443 and i can't start http. it started after i had to reboot the machine.

[Rob Crittenden]

Naor Weissmann via FreeIPA-users wrote:
> thank you!
>  the cert renewed last year, by itself it is ok and working (it is two years one). However im having issues to add servers to FreeIPA, since it is done via 443 and i can't start http. it started after i had to reboot the machine.

mod_nss can also return not found if the certificate isn't valid. See if
you can verify the cert with:

# certutil -V -u V -n Server-Cert -d /etc/httpd/alias -e -f
/etc/httpd/alias/pwdfile.txt

The certificate also needs to have the server cert EKU:

# certutil -L -d /etc/httpd/alias -n Server-Cert
...
            Name: Extended Key Usage
                TLS Web Server Authentication Certificate
...

rob

[Naor Weissmann]

thank you. both seem legit.

#certutil -V -u V -n Server-Cert -d /etc/httpd/alias -e -f /etc/httpd/alias/pwdfile.txt
certutil: certificate is valid

#certutil -L -d /etc/httpd/alias -n Server-Cert
...
Name: Extended Key Usage
                TLS Web Server Authentication Certificate
                TLS Web Client Authentication Certificate
...
naor

[Rob Crittenden]

Naor Weissmann via FreeIPA-users wrote:
> thank you. both seem legit.
>
> #certutil -V -u V -n Server-Cert -d /etc/httpd/alias -e -f /etc/httpd/alias/pwdfile.txt
> certutil: certificate is valid
>
> #certutil -L -d /etc/httpd/alias -n Server-Cert
> ...
> Name: Extended Key Usage
>                 TLS Web Server Authentication Certificate
>                 TLS Web Client Authentication Certificate
> ...

Then set LogLevel debug in nss.conf to see if you get more information
on issues.

Check ownership and permissions, look for AVCs.

rob

[Naor Weissmann]

Already on debug :(
ownership and permission set by the book.
AVCs are access vector caches ? is there a good way to check it ? its an old version so it might be an issue.
naor

[Rob Crittenden]

Naor Weissmann via FreeIPA-users wrote:
> Already on debug :(

You haven't shared the output for analysis.
> ownership and permission set by the book.

So /etc/httpd/alias/*.db is at least 0660 root:apache?
> AVCs are access vector caches ? is there a good way to check it ? its an old version so it might be an issue.

Try to start httpd then run:

# ausearch -m AVC -ts recent

rob

[Naor Weissmann]

You haven't shared the output for analysis.
> >

Had nothing to share. just two lines:
[Tue Sep 22 13:53:49 2020] [info] Using nickname Server-Cert.
[Tue Sep 22 13:53:49 2020] [error] Certificate not found: 'Server-Cert'

So /etc/httpd/alias/*.db is at least 0660 root:apache?
> > >

that is correct

# ausearch -m AVC -ts recent
> > >

i have bunch of errors there but i thought selinux in permissive will make sure nothing is blocked. 
for example i have: 
: avc:  denied  { open } for  pid=62824 comm="nss_pcache" name="secmod.db"
will try to resolve those first i guess.

[Rob Crittenden]

Naor Weissmann via FreeIPA-users wrote:
> You haven't shared the output for analysis.
> > >
>
> Had nothing to share. just two lines:
> [Tue Sep 22 13:53:49 2020] [info] Using nickname Server-Cert.
> [Tue Sep 22 13:53:49 2020] [error] Certificate not found: 'Server-Cert'

Literally nothing else? That is unexpected.
> So /etc/httpd/alias/*.db is at least 0660 root:apache?
> > > >
>
> that is correct
>
> # ausearch -m AVC -ts recent
> > > >
>
> i have bunch of errors there but i thought selinux in permissive will make sure nothing is blocked. 
> for example i have: 
> : avc:  denied  { open } for  pid=62824 comm="nss_pcache" name="secmod.db"
> will try to resolve those first i guess.

In permissive it should be fine. It could just be bad context, you can
fix with restorecon /etc/httpd/alias/*

rob

[Naor Weissmann]

I have disabled selinux, restarted the machine still same issue :(
Totally out of ideas.
Could it be related to ciphers or protocols allowed ?

[info] Configuring server for SSL protocol
[debug] nss_engine_init.c(783): NSSProtocol:  Enabling TLSv1.0
[debug] nss_engine_init.c(788): NSSProtocol:  Enabling TLSv1.1
[debug] nss_engine_init.c(793): NSSProtocol:  Enabling TLSv1.2
[debug] nss_engine_init.c(858): NSSProtocol:  [TLS 1.0] (minimum)
[debug] nss_engine_init.c(885): NSSProtocol:  [TLS 1.2] (maximum)
[debug] nss_engine_init.c(1091): NSSCipherSuite:  Configuring permitted SSL ciphers [+rsa_rc4_128_md5,+rsa_rc4_128_sha,+rsa_3des_sha,-rsa_des_sha,-rsa_rc4_40_md5,-rsa_rc2_40_md5,-rsa_null_md5,-rsa_null_sha,+fips_3des_sha,-fips_des_sha,-fortezza,-fortezza_rc4_128_sha,-fortezza_null,-rsa_des_56_sha,-rsa_rc4_56_sha,+rsa_aes_128_sha,+rsa_aes_256_sha]

[Rob Crittenden]

Naor Weissmann via FreeIPA-users wrote:
> I have disabled selinux, restarted the machine still same issue :(
> Totally out of ideas.
> Could it be related to ciphers or protocols allowed ?

No, it wouldn't affect NSS being able to find the certificate.

If you can provide the full output of the cert that might be useful.

You could try to create a new NSS database and export the cert/key from
the current one using pk12util into it to see if there is perhaps some
corruption.
> [info] Configuring server for SSL protocol
> [debug] nss_engine_init.c(783): NSSProtocol:  Enabling TLSv1.0
> [debug] nss_engine_init.c(788): NSSProtocol:  Enabling TLSv1.1
> [debug] nss_engine_init.c(793): NSSProtocol:  Enabling TLSv1.2
> [debug] nss_engine_init.c(858): NSSProtocol:  [TLS 1.0] (minimum)
> [debug] nss_engine_init.c(885): NSSProtocol:  [TLS 1.2] (maximum)
> [debug] nss_engine_init.c(1091): NSSCipherSuite:  Configuring permitted SSL ciphers [+rsa_rc4_128_md5,+rsa_rc4_128_sha,+rsa_3des_sha,-rsa_des_sha,-rsa_rc4_40_md5,-rsa_rc2_40_md5,-rsa_null_md5,-rsa_null_sha,+fips_3des_sha,-fips_des_sha,-fortezza,-fortezza_rc4_128_sha,-fortezza_null,-rsa_des_56_sha,-rsa_rc4_56_sha,+rsa_aes_128_sha,+rsa_aes_256_sha]

Oh, there is additional debug output.

rob