Draft Test Cases for Fedora Server Final Criteria

[Stephen Gallagher]

So, at today's Server WG meeting, I was asked to come up with a
high-level draft of some additional requirements to validate for Final
criteria. Most of these probably should be Beta criteria in F22, but we
didn't have them as such this time around. I'll differentiate them as
such below. It should be understood that Final validation is a superset
of Beta validation, so anything we add in that category applies to both.

== Beta Criteria ==

=== Domain Controller ===
* The Domain Controller must be capable of serving LDAP requests on port
389. This should be validated by the use of the ldapsearch tool.

* The Domain Controller must be capable of serving TLS-encrypted LDAP
requests on port 389. This should be validated by the use of the
ldapsearch tool.

* The Domain Controller must be capable of serving LDAPS (LDAP encrypted
with SSL) over port 636. This should be validated by the use of the
ldapsearch tool.

* The Domain Controller must be capable of returning LDAP and LDAPS
search results using simple auth (the -x option to ldapsearch) or
SASL/GSSAPI auth (the -Y GSSAPI option). This should be validated by the
use of the ldapsearch tool.

* The Domain Controller must be capable of serving DNS host records on
port 53. This should be validated by the use of the 'dig' tool.

=== FreeIPA Domain Client ===
* Enrolled clients must be capable of authenticating against a valid
user account using SSSD.
* Enrolled clients must honor FreeIPA HBAC rules for access-control.
* Enrolled clients must be able to change their passwords according to
the password policy specified by the FreeIPA server
* Users must be capable of performing password-less single-sign-on
between two enrolled clients using GSSAPI.



== Final Criteria ==

=== Domain Controller ===
* The Domain Controller must be capable of serving DNS SRV records for
ldap and kerberos on port 53. This is used for auto-discovery.

=== FreeIPA Domain Client ===
* When configured to use the Domain Controller for DNS services, the
domain client must be able to use DNS to discover the Domain Controller
address using SRV records.

* When configured to use FreeIPA for host-key validation, initial SSH
between domain clients should not prompt the user to accept the SSH
public key.

[Miloslav Trmač]

Hello,
I have no substantial comments so I will just nitpick…
----- Original Message -----
>  * The Domain Controller must be capable of returning LDAP and LDAPS
>  search results using simple auth (the -x option to ldapsearch) or
>  SASL/GSSAPI auth (the -Y GSSAPI option). 
Is the criterion to be capable of both
auth methods, or either one?
    Mirek

[Stephen Gallagher]

>  On Nov 5, 2014, at 6:12 AM, Miloslav Trmač <mitr(a)redhat.com&gt;
> wrote:
>  
>  Hello,
>  I have no substantial comments so I will just nitpick…
>  ----- Original Message -----
> > * The Domain Controller must be capable of returning LDAP and LDAPS
> > search results using simple auth (the -x option to ldapsearch) or
> > SASL/GSSAPI auth (the -Y GSSAPI option).
>  Is the criterion to be capable of both auth methods, or either one?
>  
Good point. The criterion is that it must succeed at both (though not in the same
command)


>     Mirek
>  _______________________________________________
>  server mailing list
>  server(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/server
>

[Stephen Gallagher]

On Tue, 2014-11-04 at 16:58 -0500, Stephen Gallagher wrote:
>  So, at today's Server WG meeting, I was asked to come up with a
>  high-level draft of some additional requirements to validate for Final
>  criteria. Most of these probably should be Beta criteria in F22, but we
>  didn't have them as such this time around. I'll differentiate them as
>  such below. It should be understood that Final validation is a superset
>  of Beta validation, so anything we add in that category applies to both.
>  
>  == Beta Criteria ==
>  
>  === Domain Controller ===
>  * The Domain Controller must be capable of serving LDAP requests on port
>  389. This should be validated by the use of the ldapsearch tool.
>  
>  * The Domain Controller must be capable of serving TLS-encrypted LDAP
>  requests on port 389. This should be validated by the use of the
>  ldapsearch tool.
>  
>  * The Domain Controller must be capable of serving LDAPS (LDAP encrypted
>  with SSL) over port 636. This should be validated by the use of the
>  ldapsearch tool.
>  
>  * The Domain Controller must be capable of returning LDAP and LDAPS
>  search results using simple auth (the -x option to ldapsearch) or
>  SASL/GSSAPI auth (the -Y GSSAPI option). This should be validated by the
>  use of the ldapsearch tool.
>  
>  * The Domain Controller must be capable of serving DNS host records on
>  port 53. This should be validated by the use of the 'dig' tool.
>  
>  === FreeIPA Domain Client ===
>  * Enrolled clients must be capable of authenticating against a valid
>  user account using SSSD.
>  * Enrolled clients must honor FreeIPA HBAC rules for access-control.
>  * Enrolled clients must be able to change their passwords according to
>  the password policy specified by the FreeIPA server
>  * Users must be capable of performing password-less single-sign-on
>  between two enrolled clients using GSSAPI.
>  
>  
>  
>  == Final Criteria ==
>  
>  === Domain Controller ===
>  * The Domain Controller must be capable of serving DNS SRV records for
>  ldap and kerberos on port 53. This is used for auto-discovery.
>  
>  === FreeIPA Domain Client ===
>  * When configured to use the Domain Controller for DNS services, the
>  domain client must be able to use DNS to discover the Domain Controller
>  address using SRV records.
>  
>  * When configured to use FreeIPA for host-key validation, initial SSH
>  between domain clients should not prompt the user to accept the SSH
>  public key. 

Any other comments on this? We enter Freeze tomorrow and I'd like for us
to have a clear view of what we're willing to block on.

[John Unland]

I guess this is just a side comment, but when doing testing last week
I really wanted to have a "Custom Role" to the software selection. I
can see this being used for specialized server software like the Yocto
Project (Embedded Linux creation tool), even more so because that
project is currently developing a web UI where you can create your
embedded OS through a web browser and also have multiple people logged
into. Of course I think this should go in F22, this is just a
thought....

On Mon, Nov 17, 2014 at 1:32 PM, Stephen Gallagher <sgallagh(a)redhat.com&gt; wrote:
>  On Tue, 2014-11-04 at 16:58 -0500, Stephen Gallagher wrote:
> > So, at today's Server WG meeting, I was asked to come up with a
> > high-level draft of some additional requirements to validate for Final
> > criteria. Most of these probably should be Beta criteria in F22, but we
> > didn't have them as such this time around. I'll differentiate them as
> > such below. It should be understood that Final validation is a superset
> > of Beta validation, so anything we add in that category applies to both.
> >
> > == Beta Criteria ==
> >
> > === Domain Controller ===
> > * The Domain Controller must be capable of serving LDAP requests on port
> > 389. This should be validated by the use of the ldapsearch tool.
> >
> > * The Domain Controller must be capable of serving TLS-encrypted LDAP
> > requests on port 389. This should be validated by the use of the
> > ldapsearch tool.
> >
> > * The Domain Controller must be capable of serving LDAPS (LDAP encrypted
> > with SSL) over port 636. This should be validated by the use of the
> > ldapsearch tool.
> >
> > * The Domain Controller must be capable of returning LDAP and LDAPS
> > search results using simple auth (the -x option to ldapsearch) or
> > SASL/GSSAPI auth (the -Y GSSAPI option). This should be validated by the
> > use of the ldapsearch tool.
> >
> > * The Domain Controller must be capable of serving DNS host records on
> > port 53. This should be validated by the use of the 'dig' tool.
> >
> > === FreeIPA Domain Client ===
> > * Enrolled clients must be capable of authenticating against a valid
> > user account using SSSD.
> > * Enrolled clients must honor FreeIPA HBAC rules for access-control.
> > * Enrolled clients must be able to change their passwords according to
> > the password policy specified by the FreeIPA server
> > * Users must be capable of performing password-less single-sign-on
> > between two enrolled clients using GSSAPI.
> >
> >
> >
> > == Final Criteria ==
> >
> > === Domain Controller ===
> > * The Domain Controller must be capable of serving DNS SRV records for
> > ldap and kerberos on port 53. This is used for auto-discovery.
> >
> > === FreeIPA Domain Client ===
> > * When configured to use the Domain Controller for DNS services, the
> > domain client must be able to use DNS to discover the Domain Controller
> > address using SRV records.
> >
> > * When configured to use FreeIPA for host-key validation, initial SSH
> > between domain clients should not prompt the user to accept the SSH
> > public key.
>
>
>  Any other comments on this? We enter Freeze tomorrow and I'd like for us
>  to have a clear view of what we're willing to block on.
>
>  _______________________________________________
>  server mailing list
>  server(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/server

[Adam Williamson]

On Mon, 2014-11-17 at 14:32 -0500, Stephen Gallagher wrote:
>  
>  
>  On Tue, 2014-11-04 at 16:58 -0500, Stephen Gallagher wrote:
>  > So, at today's Server WG meeting, I was asked to come up with a
>  > high-level draft of some additional requirements to validate for Final
>  > criteria. Most of these probably should be Beta criteria in F22, but we
>  > didn't have them as such this time around. I'll differentiate them as
>  > such below. It should be understood that Final validation is a superset
>  > of Beta validation, so anything we add in that category applies to both.
>  > 
>  > == Beta Criteria ==
>  > 
>  > === Domain Controller ===
>  > * The Domain Controller must be capable of serving LDAP requests on port
>  > 389. This should be validated by the use of the ldapsearch tool.
>  > 
>  > * The Domain Controller must be capable of serving TLS-encrypted LDAP
>  > requests on port 389. This should be validated by the use of the
>  > ldapsearch tool.
>  > 
>  > * The Domain Controller must be capable of serving LDAPS (LDAP encrypted
>  > with SSL) over port 636. This should be validated by the use of the
>  > ldapsearch tool.
>  > 
>  > * The Domain Controller must be capable of returning LDAP and LDAPS
>  > search results using simple auth (the -x option to ldapsearch) or
>  > SASL/GSSAPI auth (the -Y GSSAPI option). This should be validated by the
>  > use of the ldapsearch tool.
>  > 
>  > * The Domain Controller must be capable of serving DNS host records on
>  > port 53. This should be validated by the use of the 'dig' tool.
>  > 
>  > === FreeIPA Domain Client ===
>  > * Enrolled clients must be capable of authenticating against a valid
>  > user account using SSSD.
>  > * Enrolled clients must honor FreeIPA HBAC rules for access-control.
>  > * Enrolled clients must be able to change their passwords according to
>  > the password policy specified by the FreeIPA server
>  > * Users must be capable of performing password-less single-sign-on
>  > between two enrolled clients using GSSAPI.
>  > 
>  > 
>  > == Final Criteria ==
>  > 
>  > === Domain Controller ===
>  > * The Domain Controller must be capable of serving DNS SRV records for
>  > ldap and kerberos on port 53. This is used for auto-discovery.
>  > 
>  > === FreeIPA Domain Client ===
>  > * When configured to use the Domain Controller for DNS services, the
>  > domain client must be able to use DNS to discover the Domain Controller
>  > address using SRV records.
>  > 
>  > * When configured to use FreeIPA for host-key validation, initial SSH
>  > between domain clients should not prompt the user to accept the SSH
>  > public key.
>  
>  
>  Any other comments on this? We enter Freeze tomorrow and I'd like for us
>  to have a clear view of what we're willing to block on. 
As these were approved at a meeting I will tweak them up a bit and add
them to the criteria pages. however, I'd add a side note that they clash
a tad with the overall design philosophy of the existing criteria, which
is generally to emphasize the ultimate user/admin-visible behaviour. To
take the clearest example, consider this criterion:

"The Domain Controller must be capable of serving DNS SRV records for
ldap and kerberos on port 53. This is used for auto-discovery."

The general approach taken by the current criteria would be to write a
requirement that auto-discovery of controllers must work - i.e. focus on
the ultimate desired behaviour (the auto-discovery), not require a
specific technical behaviour that is currently used to implement it
'serve a DNS SRV record for these services on this port'). The basic
idea is that the technical implementation of desired behaviours is
likely to change more often than the desired behaviours themselves,
hence leading to more work to keep the criteria up to date if the
'describe specific technical implementation details' approach is taken.

I don't think it's a critical problem, especially for the requirements
which are specific to the domain controller role, as we want to separate
those from the release criteria per se for F22+ anyway - I just thought
I'd note it.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Adam Williamson]

On Tue, 2014-11-04 at 16:58 -0500, Stephen Gallagher wrote:

>  == Beta Criteria == 
>  === FreeIPA Domain Client ===
>  * Enrolled clients must be capable of authenticating against a valid
>  user account using SSSD. 
We already have in the Beta criteria:

* Client systems must be able to authenticate users with Kerberos

so I'm inclined to leave this one out for now.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Adam Williamson]

On Tue, 2014-11-04 at 16:58 -0500, Stephen Gallagher wrote:
>  So, at today's Server WG meeting, I was asked to come up with a
>  high-level draft of some additional requirements to validate for Final
>  criteria. Most of these probably should be Beta criteria in F22, but we
>  didn't have them as such this time around. I'll differentiate them as
>  such below. It should be understood that Final validation is a superset
>  of Beta validation, so anything we add in that category applies to both.
>  
>  == Beta Criteria ==
>  
>  === Domain Controller ===
>  * The Domain Controller must be capable of serving LDAP requests on port
>  389. This should be validated by the use of the ldapsearch tool.
>  
>  * The Domain Controller must be capable of serving TLS-encrypted LDAP
>  requests on port 389. This should be validated by the use of the
>  ldapsearch tool.
>  
>  * The Domain Controller must be capable of serving LDAPS (LDAP encrypted
>  with SSL) over port 636. This should be validated by the use of the
>  ldapsearch tool.
>  
>  * The Domain Controller must be capable of returning LDAP and LDAPS
>  search results using simple auth (the -x option to ldapsearch) or
>  SASL/GSSAPI auth (the -Y GSSAPI option). This should be validated by the
>  use of the ldapsearch tool.
>  
>  * The Domain Controller must be capable of serving DNS host records on
>  port 53. This should be validated by the use of the 'dig' tool.
>  
>  === FreeIPA Domain Client ===
>  * Enrolled clients must be capable of authenticating against a valid
>  user account using SSSD.
>  * Enrolled clients must honor FreeIPA HBAC rules for access-control.
>  * Enrolled clients must be able to change their passwords according to
>  the password policy specified by the FreeIPA server
>  * Users must be capable of performing password-less single-sign-on
>  between two enrolled clients using GSSAPI.
>  
>  
>  
>  == Final Criteria ==
>  
>  === Domain Controller ===
>  * The Domain Controller must be capable of serving DNS SRV records for
>  ldap and kerberos on port 53. This is used for auto-discovery.
>  
>  === FreeIPA Domain Client ===
>  * When configured to use the Domain Controller for DNS services, the
>  domain client must be able to use DNS to discover the Domain Controller
>  address using SRV records.
>  
>  * When configured to use FreeIPA for host-key validation, initial SSH
>  between domain clients should not prompt the user to accept the SSH
>  public key. 
I have added these criteria to the Fedora 21 pages now.

I revised the Beta page slightly:

https://fedoraproject.org/w/index.php?title=Fedora_21_Beta_Release_Criter...

to move the 'clients must be able to authenticate' criterion out of the
domain controller role requirements section (as it really is a
requirement for clients, and thus applies more widely) and into a new
section I created for domain client requirements, and explain the
circumstances in which those apply.

On the Final page, I added all the new domain controller role
requirements, following the format we established at Beta for role
requirements:

https://fedoraproject.org/w/index.php?title=Fedora_21_Final_Release_Crite...

and then created the same new 'domain client requirements' section as I
created for Beta, and added all the new client requirements to it:

https://fedoraproject.org/w/index.php?title=Fedora_21_Final_Release_Crite...

I added all the new requirements to Final, not Beta, because it would be
misleading to list them in the 21 Beta criteria when we did not in fact
test the 21 Beta against those criteria. We can move the Beta ones to
the Beta section for F22 (for the 'client requirements'; as already
noted, we should move *all* the role requirements out of the criteria
into some more sustainable framework for F22).

I tweaked the wording a bit to fit into the overall flow of the
criteria, but nothing major. 

I dropped the bits about 'this should be validated with blah' as they
are not properly part of the criteria. They're really extremely
minimalist test cases. What actually needs to happen is that we write
proper test cases for each criterion and reference them in the
References expander - you'll note that each criterion's References
section currently has FIXME as its 'Test Case'. You can see the existing
Server test cases at
https://fedoraproject.org/wiki/Template:Server_test_matrix ; we need to
write new ones following the normal layout (use, say,
https://fedoraproject.org/wiki/QA:Testcase_realmd_join_cockpit or
https://fedoraproject.org/wiki/QA:Testcase_Server_cockpit_basic as a
model), add them to that matrix, and reference them from the release
criteria. I can do some work on that tomorrow, any help anyone else can
extend in that work would be greatly appreciated.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net