Initial set of proposed release criteria for Server product

[Adam Williamson]

Hi, folks. Here's my initial dump of proposed release criteria specific
to Server. This is primarily a Server WG responsibility (per the draft
test plan -
https://fedoraproject.org/wiki/User:Adamwill/Draft_Fedora_21_test_plan -
and the discussion of same), but CCing test@ as I know some folks there
will be interested. Please ensure follow-ups *at least* go to server@
(no follow-ups only to test@.)

* It must be possible to forward system logs from one system running the
release to another using rsyslog.

* After system installation, the system firewall must be active, and the
only ports which may be open are port 22 and any ports associated with
server Roles selected during installation. [pace explicit kickstart
configuration]

* After system installation, SELinux must be enabled and in enforcing
mode. [if user does not explicitly request otherwise via cmdline or
kickstart]

* It must be possible to join the system to a FreeIPA or Active
Directory domain, and the system must respect the identity,
authentication and access control configuration provided by the domain.
[details as subnotes]

The bits in square brackets are rough notes; they indicate things that
would be expanded on via the 'expandable sub-notes' mechanism used on
the release criteria pages at present (see any of the release criteria
pages, e.g.
https://fedoraproject.org/wiki/Fedora_21_Alpha_Release_Criteria ). The
text given above would be the 'main text' for each criterion. We haven't
yet really settled the question of how exactly the presentation of the
criteria needs to be done in a Product world, but we can consider the
actual additional criteria that are required more or less independently
of how they should be presented, I think.

This set of criteria are the ones I think need to be added to 'back' the
tech spec,
https://fedoraproject.org/wiki/Server/Technical_Specification ,
*excluding* Role functionality - I expect we'll need a whole subset of
criteria for Role functionality, so I figured I can write that up as a
separate batch to keep each chunk a manageable size.

There are two areas which probably need to be captured with a criterion,
but which I couldn't really write one for yet because of vagueness in
the spec. One is problem reporting mechanisms - see my separate mail on
that topic. The other is remote system management. The spec says:

"Software updates on the Fedora Server must be possible to perform
either locally using command-line tools (e.g. yum/dnf)..."

(okay, we already cover that in current criteria)

"...or centrally by common management systems (e.g. Puppet, Chef,
Satellite, Spacewalk, OpenLMI)."

well, that's extremely broad. Do we really want to have the criteria say
"it must be possible to update a Fedora Server system via Puppet, Chef,
Satellite, Spacewalk or OpenLMI", write a test case for each, and block
releases unless all of those mechanisms work? Or do we want to focus
down a bit?

Thoughts on the proposed criteria, the remote update issue, or any other
criteria we might need beyond the existing 'core' criteria and the ones
for Role functionality would be great! Thanks folks.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Miloslav Trmač]

Hello,
2014-06-07 0:55 GMT+02:00 Adam Williamson <awilliam(a)redhat.com&gt;:

>  The other is remote system management. The spec says:
>
>  "Software updates on the Fedora Server must be possible to perform
>  either locally using command-line tools (e.g. yum/dnf)..."
>
>  (okay, we already cover that in current criteria)
>
>  "...or centrally by common management systems (e.g. Puppet, Chef,
>  Satellite, Spacewalk, OpenLMI)."
>
>  well, that's extremely broad. Do we really want to have the criteria say
>  "it must be possible to update a Fedora Server system via Puppet, Chef,
>  Satellite, Spacewalk or OpenLMI", write a test case for each, and block
>  releases unless all of those mechanisms work? Or do we want to focus
>  down a bit?
>  
I don’t recall precisely; AFAICT this is mainly a “we shall not break what
works” criterion, e.g. that Server should not add a mandatory interactivity
requirement for updates that would make it impossible to use some of these
tools.
    Mirek

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/09/2014 08:26 AM, Miloslav Trmač wrote:
>  Hello, 2014-06-07 0:55 GMT+02:00 Adam Williamson
>  <awilliam(a)redhat.com <mailto:[e-mail]>>:
>  
>  The other is remote system management. The spec says:
>  
>  "Software updates on the Fedora Server must be possible to perform 
>  either locally using command-line tools (e.g. yum/dnf)..."
>  
>  (okay, we already cover that in current criteria)
>  
>  "...or centrally by common management systems (e.g. Puppet, Chef, 
>  Satellite, Spacewalk, OpenLMI)."
>  
>  well, that's extremely broad. Do we really want to have the
>  criteria say "it must be possible to update a Fedora Server system
>  via Puppet, Chef, Satellite, Spacewalk or OpenLMI", write a test
>  case for each, and block releases unless all of those mechanisms
>  work? Or do we want to focus down a bit?
>  
>  
>  I don’t recall precisely; AFAICT this is mainly a “we shall not
>  break what works” criterion, e.g. that Server should not add a
>  mandatory interactivity requirement for updates that would make it
>  impossible to use some of these tools. 
Yes, I'm pretty sure that was the intent of that portion: mainly that
we commit to not actively hampering the efforts of those (and similar)
projects from working with Fedora Server (even if we eventually
support and provide a "better" option).

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlOV08QACgkQeiVVYja6o6NXvwCaAzX0ogXMG9zeGjxM4QT9XZ8v
> +RYAoIPsWWMZOUoKpICoAw2Sb+NZvi4t
> =pCeb
> -----END PGP SIGNATURE-----

[Adam Williamson]

On Mon, 2014-06-09 at 11:33 -0400, Stephen Gallagher wrote:
>  On 06/09/2014 08:26 AM, Miloslav Trmač wrote:
>  > Hello, 2014-06-07 0:55 GMT+02:00 Adam Williamson
>  > <awilliam(a)redhat.com <mailto:[e-mail]>>:
>  > 
>  > The other is remote system management. The spec says:
>  > 
>  > "Software updates on the Fedora Server must be possible to perform 
>  > either locally using command-line tools (e.g. yum/dnf)..."
>  > 
>  > (okay, we already cover that in current criteria)
>  > 
>  > "...or centrally by common management systems (e.g. Puppet, Chef, 
>  > Satellite, Spacewalk, OpenLMI)."
>  > 
>  > well, that's extremely broad. Do we really want to have the
>  > criteria say "it must be possible to update a Fedora Server system
>  > via Puppet, Chef, Satellite, Spacewalk or OpenLMI", write a test
>  > case for each, and block releases unless all of those mechanisms
>  > work? Or do we want to focus down a bit?
>  > 
>  > 
>  > I don’t recall precisely; AFAICT this is mainly a “we shall not
>  > break what works” criterion, e.g. that Server should not add a
>  > mandatory interactivity requirement for updates that would make it
>  > impossible to use some of these tools.
>  
>  Yes, I'm pretty sure that was the intent of that portion: mainly that
>  we commit to not actively hampering the efforts of those (and similar)
>  projects from working with Fedora Server (even if we eventually
>  support and provide a "better" option). 
OK. I can write a criterion based on that interpretation fine; would it
be worth rewording the tech spec a bit, perhaps something like this:

"Fedora Server will provide a command line utility for performing system
updates and will ensure that updates can always be run
non-interactively. Server will aim to maintain compatibility with
management systems such as Puppet, Chef, Satellite, Spacewalk and
OpenLMI for remote update deployment."

(it occurs to me that we don't specify anything about repository
configuration or package signing there, I guess they're stuff we kind of
assume is baked into Fedora...)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/09/2014 08:05 PM, Adam Williamson wrote:
>  On Mon, 2014-06-09 at 11:33 -0400, Stephen Gallagher wrote:
> > On 06/09/2014 08:26 AM, Miloslav Trmač wrote:
> >> Hello, 2014-06-07 0:55 GMT+02:00 Adam Williamson 
> >> <awilliam(a)redhat.com <mailto:[e-mail]>>:
> >> 
> >> The other is remote system management. The spec says:
> >> 
> >> "Software updates on the Fedora Server must be possible to
> >> perform either locally using command-line tools (e.g.
> >> yum/dnf)..."
> >> 
> >> (okay, we already cover that in current criteria)
> >> 
> >> "...or centrally by common management systems (e.g. Puppet,
> >> Chef, Satellite, Spacewalk, OpenLMI)."
> >> 
> >> well, that's extremely broad. Do we really want to have the 
> >> criteria say "it must be possible to update a Fedora Server
> >> system via Puppet, Chef, Satellite, Spacewalk or OpenLMI",
> >> write a test case for each, and block releases unless all of
> >> those mechanisms work? Or do we want to focus down a bit?
> >> 
> >> 
> >> I don’t recall precisely; AFAICT this is mainly a “we shall
> >> not break what works” criterion, e.g. that Server should not
> >> add a mandatory interactivity requirement for updates that
> >> would make it impossible to use some of these tools.
> > 
> > Yes, I'm pretty sure that was the intent of that portion: mainly
> > that we commit to not actively hampering the efforts of those
> > (and similar) projects from working with Fedora Server (even if
> > we eventually support and provide a "better" option).
>  
>  OK. I can write a criterion based on that interpretation fine;
>  would it be worth rewording the tech spec a bit, perhaps something
>  like this:
>  
>  "Fedora Server will provide a command line utility for performing
>  system updates and will ensure that updates can always be run 
>  non-interactively. Server will aim to maintain compatibility with 
>  management systems such as Puppet, Chef, Satellite, Spacewalk and 
>  OpenLMI for remote update deployment."
>   
That seems reasonable to me. +1

>  (it occurs to me that we don't specify anything about repository
>
>  configuration or package signing there, I guess they're stuff we
>  kind of assume is baked into Fedora...)
>   
Yeah, that's a duty I'm perfectly comfortable leaving in the Base
Design jurisdiction.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEUEARECAAYFAlOW5xwACgkQeiVVYja6o6MglQCgrUnwnilME720GNW5ZgLwSJaT
> 568Al2m2aHN760V7L4q0jAhDIPisSTY=
> =HEYh
> -----END PGP SIGNATURE-----

[Adam Williamson]

On Mon, 2014-06-09 at 17:05 -0700, Adam Williamson wrote:
>  On Mon, 2014-06-09 at 11:33 -0400, Stephen Gallagher wrote:
>  > On 06/09/2014 08:26 AM, Miloslav Trmač wrote:
>  > > Hello, 2014-06-07 0:55 GMT+02:00 Adam Williamson
>  > > <awilliam(a)redhat.com <mailto:[e-mail]>>:
>  > > 
>  > > The other is remote system management. The spec says:
>  > > 
>  > > "Software updates on the Fedora Server must be possible to perform 
>  > > either locally using command-line tools (e.g. yum/dnf)..."
>  > > 
>  > > (okay, we already cover that in current criteria)
>  > > 
>  > > "...or centrally by common management systems (e.g. Puppet, Chef, 
>  > > Satellite, Spacewalk, OpenLMI)."
>  > > 
>  > > well, that's extremely broad. Do we really want to have the
>  > > criteria say "it must be possible to update a Fedora Server system
>  > > via Puppet, Chef, Satellite, Spacewalk or OpenLMI", write a test
>  > > case for each, and block releases unless all of those mechanisms
>  > > work? Or do we want to focus down a bit?
>  > > 
>  > > 
>  > > I don’t recall precisely; AFAICT this is mainly a “we shall not
>  > > break what works” criterion, e.g. that Server should not add a
>  > > mandatory interactivity requirement for updates that would make it
>  > > impossible to use some of these tools.
>  > 
>  > Yes, I'm pretty sure that was the intent of that portion: mainly that
>  > we commit to not actively hampering the efforts of those (and similar)
>  > projects from working with Fedora Server (even if we eventually
>  > support and provide a "better" option).
>  
>  OK. I can write a criterion based on that interpretation fine; would it
>  be worth rewording the tech spec a bit, perhaps something like this:
>  
>  "Fedora Server will provide a command line utility for performing system
>  updates and will ensure that updates can always be run
>  non-interactively. Server will aim to maintain compatibility with
>  management systems such as Puppet, Chef, Satellite, Spacewalk and
>  OpenLMI for remote update deployment."
>  
>  (it occurs to me that we don't specify anything about repository
>  configuration or package signing there, I guess they're stuff we kind of
>  assume is baked into Fedora...) 
Having thought about this a bit more, I think it possibly doesn't need a
release criterion. It seems quite unlikely that some kind of *bug* is
going to introduce interactivity into the update process; if it
happened, it'd be a design/policy issue, not something to handle via
what's a fairly bug-centred process. OK, maybe that's a bit squishy, but
it kinda makes sense in my head; it just doesn't seem to fit into my
release criteria mental box, every time I draft a criterion for it it
comes out sounding...odd.

Can anyone see something specific here which should be called out as a
release criterion and have a particular validation test added? Something
specific that we can (and need to) test at each release milestone?
Beyond the existing generic criteria/tests? I can't, quite. So unless
someone else can, I'll leave it out.

We could, I guess, promote one or a few particular remote management
system(s) to 'formally supported' status, and have a criterion / test
case for "it must be possible to perform an automated remote update of
the system from Puppet [or Chef or Satellite or whichever one(s) we
pick]". That's the thing that feels like a criterion / test case to me,
if anything. Thoughts?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Adam Williamson]

On Fri, 2014-06-06 at 15:55 -0700, Adam Williamson wrote:

More criteria considerations:

>  * It must be possible to forward system logs from one system running
> the
>  release to another using rsyslog.
>  
>  * After system installation, the system firewall must be active, and the
>  only ports which may be open are port 22 and any ports associated with
>  server Roles selected during installation. [pace explicit kickstart
>  configuration] 
If you look at these two - they kind of mutually contradict a bit. We
want log forwarding to work, but we're explicitly requiring that any
port likely to be used for it be firewalled?

Would "log server" be a viable role, hence you'd deploy your "log
server
role" and the appropriate firewall ports would be opened as a part of
that? That could be a long term solution (or even short term, if someone
wanted to throw that role together).

Otherwise I can just write an exception for the firewall port into the
system logging criterion.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Simo Sorce]

On Wed, 2014-06-11 at 16:28 -0700, Adam Williamson wrote:
>  On Fri, 2014-06-06 at 15:55 -0700, Adam Williamson wrote:
>  
>  More criteria considerations:
>  
>  > * It must be possible to forward system logs from one system running the
>  > release to another using rsyslog.
>  > 
>  > * After system installation, the system firewall must be active, and the
>  > only ports which may be open are port 22 and any ports associated with
>  > server Roles selected during installation. [pace explicit kickstart
>  > configuration]
>  
>  If you look at these two - they kind of mutually contradict a bit. We
>  want log forwarding to work, but we're explicitly requiring that any
>  port likely to be used for it be firewalled? 
I do not see a contradiction, the point say you must be able to forward
logs, not to receive logs. Of course you need a log server, but that
could be anything, Fedora or not, and clearly you do need to configure
it after installation (including any client you may want to get logs
from).

>  Would "log server" be a viable role, hence you'd deploy
> your "log server
>  role" and the appropriate firewall ports would be opened as a part of
>  that? That could be a long term solution (or even short term, if someone
>  wanted to throw that role together). 
A log server may be an appropriate role, but for the future.

>  Otherwise I can just write an exception for the firewall port into
> the
>  system logging criterion. 
No exception is needed in my opinion.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Adam Williamson]

On Wed, 2014-06-11 at 21:53 -0400, Simo Sorce wrote:
>  On Wed, 2014-06-11 at 16:28 -0700, Adam Williamson wrote:
>  > On Fri, 2014-06-06 at 15:55 -0700, Adam Williamson wrote:
>  > 
>  > More criteria considerations:
>  > 
>  > > * It must be possible to forward system logs from one system running the
>  > > release to another using rsyslog.
>  > > 
>  > > * After system installation, the system firewall must be active, and the
>  > > only ports which may be open are port 22 and any ports associated with
>  > > server Roles selected during installation. [pace explicit kickstart
>  > > configuration]
>  > 
>  > If you look at these two - they kind of mutually contradict a bit. We
>  > want log forwarding to work, but we're explicitly requiring that any
>  > port likely to be used for it be firewalled?
>  
>  I do not see a contradiction, the point say you must be able to forward
>  logs, not to receive logs. Of course you need a log server, but that
>  could be anything, Fedora or not, and clearly you do need to configure
>  it after installation (including any client you may want to get logs
>  from). 
The intent of the wording was actually to cover both functions, as I
assumed we'd want that. However, now I see it's not entirely clear. "one
system running the release to another" is meant to mean that *both
systems in question* are running the Fedora release to which the
criterion is being applied, but now I see that's not entirely clear.

>  
>  > Would "log server" be a viable role, hence you'd deploy your "log
> server
>  > role" and the appropriate firewall ports would be opened as a part of
>  > that? That could be a long term solution (or even short term, if someone
>  > wanted to throw that role together).
>  
>  A log server may be an appropriate role, but for the future.
>  
>  > Otherwise I can just write an exception for the firewall port into the
>  > system logging criterion.
>  
>  No exception is needed in my opinion.
>  
>  Simo.
>   
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/11/2014 07:28 PM, Adam Williamson wrote:
>  On Fri, 2014-06-06 at 15:55 -0700, Adam Williamson wrote:
>  
>  More criteria considerations:
>  
> > * It must be possible to forward system logs from one system
> > running the release to another using rsyslog.
> > 
> > * After system installation, the system firewall must be active,
> > and the only ports which may be open are port 22 and any ports
> > associated with server Roles selected during installation. [pace
> > explicit kickstart configuration]
>  
>  If you look at these two - they kind of mutually contradict a bit.
>  We want log forwarding to work, but we're explicitly requiring that
>  any port likely to be used for it be firewalled?
>   
It should probably read "the only incoming ports which may be open".
Log forwarding is an outgoing operation.


Also, this is only the default right-after-installation state. If
someone wants to use a machine as an rsyslog aggregation host, they're
going to need to do other configuration; asking them to call
firewall-cmd to open the external port is a reasonable request. It
shouldn't be open by default.


>  Would "log server" be a viable role, hence you'd deploy
> your "log
>  server role" and the appropriate firewall ports would be opened as
>  a part of that? That could be a long term solution (or even short
>  term, if someone wanted to throw that role together). 
Yes, this is one I want to strongly consider for F22.


>  
>  Otherwise I can just write an exception for the firewall port into
>  the system logging criterion.
>   
See above.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlOZuqsACgkQeiVVYja6o6MCJQCfYRvCzIyP+12XlkveMMWg3Cq4
> 1D4AoKoH7yEGTTWgD3S2KJNLiDbUUHQq
> =da7/
> -----END PGP SIGNATURE-----

[Adam Williamson]

On Thu, 2014-06-12 at 10:35 -0400, Stephen Gallagher wrote:

>  > If you look at these two - they kind of mutually contradict a
> bit.
>  > We want log forwarding to work, but we're explicitly requiring that
>  > any port likely to be used for it be firewalled?
>  > 
>  
>  It should probably read "the only incoming ports which may be open".
>  Log forwarding is an outgoing operation. 
That's how it reads (more or less) in the second draft, but I was
considering the *server* system (see other mail about how this criterion
was written to require both client and server functions to work).

>  Also, this is only the default right-after-installation state. If
>  someone wants to use a machine as an rsyslog aggregation host, they're
>  going to need to do other configuration; asking them to call
>  firewall-cmd to open the external port is a reasonable request. It
>  shouldn't be open by default. 
Sure - but I meant writing an 'exception' into the rsyslog criterion,
not the firewall one. I did it in the second draft, anyway, so see if
that makes sense to you.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

Second draft of server criteria, questions (was Re: Initial set of proposed release criteria for Server product)

[Adam Williamson]

On Fri, 2014-06-06 at 15:55 -0700, Adam Williamson wrote:
>  Hi, folks. Here's my initial dump of proposed release criteria
> specific
>  to Server. This is primarily a Server WG responsibility (per the draft
>  test plan -
>  https://fedoraproject.org/wiki/User:Adamwill/Draft_Fedora_21_test_plan -
>  and the discussion of same), but CCing test@ as I know some folks there
>  will be interested. Please ensure follow-ups *at least* go to server@
>  (no follow-ups only to test@.) 
I've now cleaned up three of the proposed criteria and posted them in
the 'stock format' here:

https://fedoraproject.org/wiki/User:Adamwill/Draft_server_release_criteria

I did some refinements as I wrote. Things tend to occur to you as you
write release criteria. Do check over the notes. Particularly, there's
this, from the firewall criterion:

"Should kickstart firewall options conflict with the settings that would
otherwise result from a deployed server role, the kickstart options must
take precedence."

that seems to make sense to me, but assuming everyone agrees that's how
it should be, we need to design the anaconda server role integration in
that way. If everyone thinks I'm nuts and we should resolve conflicts
the other way around...we need to do the implementation that way around.
It's something we need to take into account, is what I'm sayin'. :)

Also note "the system firewall must be active on all non-loopback
interfaces" and "The only ports which may be open to incoming traffic" -
I think those are correct/appropriate clarifications. Also note
"Supported install-time firewall configuration options must work
correctly." - I think it's reasonable to require that, but it does
introduce a moderate burden of testing and installer functionality. Full
testing might involve four install runs. One 'clean' one, one with just
a server role configured, one with a big set of kickstart firewall
directives, one with both server role and kickstart firewall directives
to test the conflict case. We could do a milestone split; maybe the
'port 22 only on clean install' part at Alpha, 'server role' bits at
Beta, and 'kickstart and role/kickstart conflict' parts at Final, or
something like that?

You'll note one criterion missing, because I spotted a rather big
ambiguity. It's the remote auth configuration one. The tech spec says:

"The Fedora Server is expected to nearly always be configured for
'centrally-managed' user information; it must be possible to configure
it to rely on a directory service for this information. Fedora Server
will provide and support the realmd project for joining FreeIPA and
Active Directory domains automatically. Interacting with other identity
sources will remain a manual configuration effort."

What it never says is whether this is expected to work *at install time*
or post-install. My guess would be that we'd want to have install time
configuration of this, but I wanted to clarify it before writing it into
the criteria.

Note that this is *not currently the case*. anaconda does not have any
remote auth configuration support of which I'm aware at present. (I'm
kinda surprised it wasn't considered a blocker for RHEL 7, in all
honesty, but hey, RHEL ain't my beat). So if we wanted to block on that,
we'd need to work out a plan with anaconda devs to have it implemented,
ideally by Alpha.

Thoughts on all the above? Thanks!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

Second draft of server criteria, questions (was Re: Initial set of proposed release criteria for Server product)

[Simo Sorce]

On Wed, 2014-06-11 at 17:14 -0700, Adam Williamson wrote:
>  On Fri, 2014-06-06 at 15:55 -0700, Adam Williamson wrote:
>  > Hi, folks. Here's my initial dump of proposed release criteria specific
>  > to Server. This is primarily a Server WG responsibility (per the draft
>  > test plan -
>  > https://fedoraproject.org/wiki/User:Adamwill/Draft_Fedora_21_test_plan -
>  > and the discussion of same), but CCing test@ as I know some folks there
>  > will be interested. Please ensure follow-ups *at least* go to server@
>  > (no follow-ups only to test@.)
>  
>  I've now cleaned up three of the proposed criteria and posted them in
>  the 'stock format' here:
>  
>  https://fedoraproject.org/wiki/User:Adamwill/Draft_server_release_criteria
>  
>  I did some refinements as I wrote. Things tend to occur to you as you
>  write release criteria. Do check over the notes. Particularly, there's
>  this, from the firewall criterion:
>  
>  "Should kickstart firewall options conflict with the settings that would
>  otherwise result from a deployed server role, the kickstart options must
>  take precedence."
>  
>  that seems to make sense to me, but assuming everyone agrees that's how
>  it should be, we need to design the anaconda server role integration in
>  that way. If everyone thinks I'm nuts and we should resolve conflicts
>  the other way around...we need to do the implementation that way around.
>  It's something we need to take into account, is what I'm sayin'. :) 
uhmm are firewall rules aplied at *install* time for roles or at
configuration time ?
If so does configuration override kickstart defaults ?

>  Also note "the system firewall must be active on all
> non-loopback
>  interfaces" and "The only ports which may be open to incoming traffic" -
>  I think those are correct/appropriate clarifications. Also note
>  "Supported install-time firewall configuration options must work
>  correctly." - I think it's reasonable to require that, but it does
>  introduce a moderate burden of testing and installer functionality. Full
>  testing might involve four install runs. One 'clean' one, one with just
>  a server role configured, one with a big set of kickstart firewall
>  directives, one with both server role and kickstart firewall directives
>  to test the conflict case. We could do a milestone split; maybe the
>  'port 22 only on clean install' part at Alpha, 'server role' bits at
>  Beta, and 'kickstart and role/kickstart conflict' parts at Final, or
>  something like that? 
Sounds reasonable but I am not entirely convinced it makes sense to have
kickstart defined firewall rules override the roles. The roles do have
an interface to tell whether you want to apply firewall rules or not, so
if you explicitly ask for firewall to be poked then we have
contradicting rules, and I think the role should win in that case.

>  You'll note one criterion missing, because I spotted a rather
> big
>  ambiguity. It's the remote auth configuration one. The tech spec says:
>  
>  "The Fedora Server is expected to nearly always be configured for
>  'centrally-managed' user information; it must be possible to configure
>  it to rely on a directory service for this information. Fedora Server
>  will provide and support the realmd project for joining FreeIPA and
>  Active Directory domains automatically. Interacting with other identity
>  sources will remain a manual configuration effort."
>  
>  What it never says is whether this is expected to work *at install time*
>  or post-install. My guess would be that we'd want to have install time
>  configuration of this, but I wanted to clarify it before writing it into
>  the criteria. 
Does it really matter if it is install or post-install ?

>  Note that this is *not currently the case*. anaconda does not have
> any
>  remote auth configuration support of which I'm aware at present. 
It used to, has it been evicted ?

>  (I'm kinda surprised it wasn't considered a blocker for RHEL
> 7, in all
>  honesty, but hey, RHEL ain't my beat). So if we wanted to block on that,
>  we'd need to work out a plan with anaconda devs to have it implemented,
>  ideally by Alpha.
>  
>  Thoughts on all the above? Thanks! 
I do not think we should make it a blocker, however I think the user
MUST NOT be forced to create a first user in this case.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

Second draft of server criteria, questions (was Re: Initial set of proposed release criteria for Server product)

[Miloslav Trmač]

2014-06-12 2:14 GMT+02:00 Adam Williamson <awilliam(a)redhat.com&gt;:

>  "Should kickstart firewall options conflict with the settings
> that would
>  otherwise result from a deployed server role, the kickstart options must
>  take precedence."
>
>  that seems to make sense to me 

Yes, assuming you are talking about an explicit kickstart “role deploy”
command and kickstart “firewall” command.  Within %post and post-install
actions, simply the last edit should win, whether it is a manual command or
a role deploy.


>  Note that this is *not currently the case*. anaconda does not have
> any
>  remote auth configuration support of which I'm aware at present. 
https://fedoraproject.org/wiki/Anaconda/Kickstart#realm ?  (Possibly in
combination with
https://fedoraproject.org/wiki/Anaconda/Kickstart#auth_or_authconfig , I’m
not sure.)
   Mirek

Second draft of server criteria, questions (was Re: Initial set of proposed release criteria for Server product)

[Adam Williamson]

On Thu, 2014-06-12 at 16:22 +0200, Miloslav Trmač wrote:
>  2014-06-12 2:14 GMT+02:00 Adam Williamson
> <awilliam(a)redhat.com&gt;:
>  
>  > "Should kickstart firewall options conflict with the settings that would
>  > otherwise result from a deployed server role, the kickstart options must
>  > take precedence."
>  >
>  > that seems to make sense to me
>  
>  
>  Yes, assuming you are talking about an explicit kickstart “role deploy”
>  command and kickstart “firewall” command. 
Yes, that's the intent. Stuff in %post is generally always 'best
effort', so far as I'm concerned.

>  > Note that this is *not currently the case*. anaconda does not
> have any
>  > remote auth configuration support of which I'm aware at present.
>  
>  https://fedoraproject.org/wiki/Anaconda/Kickstart#realm ?  (Possibly in
>  combination with
>  https://fedoraproject.org/wiki/Anaconda/Kickstart#auth_or_authconfig , I’m
>  not sure.) 
Sorry, yes - I meant to say that it's not possible to do it
*interactively* at present.

(This is theoretically a regression from oldUI, which had an interface
for it, but AIUI the oldUI interface rarely actually worked...)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

Second draft of server criteria, questions (was Re: Initial set of proposed release criteria for Server product)

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/11/2014 08:14 PM, Adam Williamson wrote:
>  On Fri, 2014-06-06 at 15:55 -0700, Adam Williamson wrote:
> > Hi, folks. Here's my initial dump of proposed release criteria
> > specific to Server. This is primarily a Server WG responsibility
> > (per the draft test plan - 
> > https://fedoraproject.org/wiki/User:Adamwill/Draft_Fedora_21_test_plan
> > - and the discussion of same), but CCing test@ as I know some
> > folks there will be interested. Please ensure follow-ups *at
> > least* go to server@ (no follow-ups only to test@.)
>  
>  I've now cleaned up three of the proposed criteria and posted them
>  in the 'stock format' here:
>  
>  https://fedoraproject.org/wiki/User:Adamwill/Draft_server_release_criteria
>
>   I did some refinements as I wrote. Things tend to occur to you as
>  you write release criteria. Do check over the notes. Particularly,
>  there's this, from the firewall criterion:
>  
>  "Should kickstart firewall options conflict with the settings that
>  would otherwise result from a deployed server role, the kickstart
>  options must take precedence."
>  
>  that seems to make sense to me, but assuming everyone agrees that's
>  how it should be, we need to design the anaconda server role
>  integration in that way. If everyone thinks I'm nuts and we should
>  resolve conflicts the other way around...we need to do the
>  implementation that way around. It's something we need to take into
>  account, is what I'm sayin'. :)
>   
That would be difficult to do, as the current expectation is that
Roles will be configured as part of the first-boot environment, after
Anaconda has concluded and the system has rebooted.

I think we should not make "resolution order of firewall conflicts" a
release criterion.


>  Also note "the system firewall must be active on all
> non-loopback 
>  interfaces" and "The only ports which may be open to incoming
>  traffic" - I think those are correct/appropriate clarifications.
>  Also note "Supported install-time firewall configuration options
>  must work correctly." - I think it's reasonable to require that,
>  but it does introduce a moderate burden of testing and installer
>  functionality. Full testing might involve four install runs. One
>  'clean' one, one with just a server role configured, one with a big
>  set of kickstart firewall directives, one with both server role and
>  kickstart firewall directives to test the conflict case. We could
>  do a milestone split; maybe the 'port 22 only on clean install'
>  part at Alpha, 'server role' bits at Beta, and 'kickstart and
>  role/kickstart conflict' parts at Final, or something like that?
>   
Manual firewall rules will *always* throw a wrench in the mix. I'd
argue that we should only commit to testing 1) clean state, 2) clean
state + role ports, 3) manually-specified rules only

I think testing Role ports + manually-configured is likely to be a
mine-field that will never have a p


>  You'll note one criterion missing, because I spotted a rather big
>
>  ambiguity. It's the remote auth configuration one. The tech spec
>  says:
>  
>  "The Fedora Server is expected to nearly always be configured for 
>  'centrally-managed' user information; it must be possible to
>  configure it to rely on a directory service for this information.
>  Fedora Server will provide and support the realmd project for
>  joining FreeIPA and Active Directory domains automatically.
>  Interacting with other identity sources will remain a manual
>  configuration effort."
>  
>  What it never says is whether this is expected to work *at install
>  time* or post-install. My guess would be that we'd want to have
>  install time configuration of this, but I wanted to clarify it
>  before writing it into the criteria.
>  
>  Note that this is *not currently the case*. anaconda does not have
>  any remote auth configuration support of which I'm aware at
>  present. (I'm kinda surprised it wasn't considered a blocker for
>  RHEL 7, in all honesty, but hey, RHEL ain't my beat). So if we
>  wanted to block on that, we'd need to work out a plan with anaconda
>  devs to have it implemented, ideally by Alpha.
>  
>  Thoughts on all the above? Thanks!
>   
This is available today with realmd's anaconda plugin (which is also
present in RHEL 7.0 final). I'm not sure if there's a graphical
solution at present; I'll need to spin up a RHEL 7 VM and check it. I
know it works in kickstart though.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlOZvNMACgkQeiVVYja6o6OA0wCeMwE4+sSJBdoPGZqsAasjdYI+
> +RQAnAmp0MzpsAYPGILNam/KWJ3XuGiF
> =DmPs
> -----END PGP SIGNATURE-----

Second draft of server criteria, questions (was Re: Initial set of proposed release criteria for Server product)

[Adam Williamson]

On Thu, 2014-06-12 at 10:44 -0400, Stephen Gallagher wrote:

>  That would be difficult to do, as the current expectation is that
>  Roles will be configured as part of the first-boot environment, after
>  Anaconda has concluded and the system has rebooted.
>  
>  I think we should not make "resolution order of firewall conflicts" a
>  release criterion. 
>  I think testing Role ports + manually-configured is likely to be a
>  mine-field that will never have a p 
A pea? :)

That's all fine by me if others agree. I can dial back the draft.

>  > You'll note one criterion missing, because I spotted a
> rather big 
>  > ambiguity. It's the remote auth configuration one. The tech spec
>  > says:
>  > 
>  > "The Fedora Server is expected to nearly always be configured for 
>  > 'centrally-managed' user information; it must be possible to
>  > configure it to rely on a directory service for this information.
>  > Fedora Server will provide and support the realmd project for
>  > joining FreeIPA and Active Directory domains automatically.
>  > Interacting with other identity sources will remain a manual
>  > configuration effort."
>  > 
>  > What it never says is whether this is expected to work *at install
>  > time* or post-install. My guess would be that we'd want to have
>  > install time configuration of this, but I wanted to clarify it
>  > before writing it into the criteria.
>  > 
>  > Note that this is *not currently the case*. anaconda does not have
>  > any remote auth configuration support of which I'm aware at
>  > present. (I'm kinda surprised it wasn't considered a blocker for
>  > RHEL 7, in all honesty, but hey, RHEL ain't my beat). So if we
>  > wanted to block on that, we'd need to work out a plan with anaconda
>  > devs to have it implemented, ideally by Alpha.
>  > 
>  > Thoughts on all the above? Thanks!
>  
>  This is available today with realmd's anaconda plugin (which is also
>  present in RHEL 7.0 final). I'm not sure if there's a graphical
>  solution at present; I'll need to spin up a RHEL 7 VM and check it. I
>  know it works in kickstart though. 
Yeah, sorry, as noted in my other mail I meant that it is not possible
interactively.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

Second draft of server criteria, questions (was Re: Initial set of proposed release criteria for Server product)

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/12/2014 11:37 AM, Adam Williamson wrote:
>  On Thu, 2014-06-12 at 10:44 -0400, Stephen Gallagher wrote:
>  
> > That would be difficult to do, as the current expectation is
> > that Roles will be configured as part of the first-boot
> > environment, after Anaconda has concluded and the system has
> > rebooted.
> > 
> > I think we should not make "resolution order of firewall
> > conflicts" a release criterion.
>  
> > I think testing Role ports + manually-configured is likely to be
> > a mine-field that will never have a p
>  
>  A pea? :)
>   
Strange, I definitely typed "erfect solution." after that, but somehow
it got lost.


>  That's all fine by me if others agree. I can dial back the
> draft.
>  
> >> You'll note one criterion missing, because I spotted a rather
> >> big ambiguity. It's the remote auth configuration one. The tech
> >> spec says:
> >> 
> >> "The Fedora Server is expected to nearly always be configured
> >> for 'centrally-managed' user information; it must be possible
> >> to configure it to rely on a directory service for this
> >> information. Fedora Server will provide and support the realmd
> >> project for joining FreeIPA and Active Directory domains
> >> automatically. Interacting with other identity sources will
> >> remain a manual configuration effort."
> >> 
> >> What it never says is whether this is expected to work *at
> >> install time* or post-install. My guess would be that we'd want
> >> to have install time configuration of this, but I wanted to
> >> clarify it before writing it into the criteria.
> >> 
> >> Note that this is *not currently the case*. anaconda does not
> >> have any remote auth configuration support of which I'm aware
> >> at present. (I'm kinda surprised it wasn't considered a blocker
> >> for RHEL 7, in all honesty, but hey, RHEL ain't my beat). So if
> >> we wanted to block on that, we'd need to work out a plan with
> >> anaconda devs to have it implemented, ideally by Alpha.
> >> 
> >> Thoughts on all the above? Thanks!
> > 
> > This is available today with realmd's anaconda plugin (which is
> > also present in RHEL 7.0 final). I'm not sure if there's a
> > graphical solution at present; I'll need to spin up a RHEL 7 VM
> > and check it. I know it works in kickstart though.
>  
>  Yeah, sorry, as noted in my other mail I meant that it is not
>  possible interactively.
>   
Interactive hasn't been our top priority for the first release
(particularly since our common-case involves headless servers). I'd
definitely keep that off the requirement list for F21 at least.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlOZyaQACgkQeiVVYja6o6OfzQCdFhSmucxiIt0B7uDP7fBm8m96
> o+UAnAprdeED/EvtQ1JEmQdiQ9/mtTo+
> =Duth
> -----END PGP SIGNATURE-----

Second draft of server criteria, questions (was Re: Initial set of proposed release criteria for Server product)

[Adam Williamson]

On Wed, 2014-06-11 at 17:14 -0700, Adam Williamson wrote:
>  On Fri, 2014-06-06 at 15:55 -0700, Adam Williamson wrote:
>  > Hi, folks. Here's my initial dump of proposed release criteria specific
>  > to Server. This is primarily a Server WG responsibility (per the draft
>  > test plan -
>  > https://fedoraproject.org/wiki/User:Adamwill/Draft_Fedora_21_test_plan -
>  > and the discussion of same), but CCing test@ as I know some folks there
>  > will be interested. Please ensure follow-ups *at least* go to server@
>  > (no follow-ups only to test@.)
>  
>  I've now cleaned up three of the proposed criteria and posted them in
>  the 'stock format' here:
>  
>  https://fedoraproject.org/wiki/User:Adamwill/Draft_server_release_criteria 
I've revised this page again today based on the feedback received so
far. Notably, I've tried to make the requirement for both ends of log
forwarding to work more clear - I'm still assuming this is actually what
we want, if we only want to mandate client not server, someone speak up.
I added the remote authentication criterion, with a requirement that it
be possible at install time but a stipulation that only being possible
non-interactively (via kickstart) is fine. I removed the explicit
requirement as regards how conflicts between Role and user-specified
firewall config should be resolved.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

Second draft of server criteria, questions (was Re: Initial set of proposed release criteria for Server product)

[Adam Williamson]

On Thu, 2014-06-12 at 19:39 -0700, Adam Williamson wrote:
>  On Wed, 2014-06-11 at 17:14 -0700, Adam Williamson wrote:
>  > On Fri, 2014-06-06 at 15:55 -0700, Adam Williamson wrote:
>  > > Hi, folks. Here's my initial dump of proposed release criteria specific
>  > > to Server. This is primarily a Server WG responsibility (per the draft
>  > > test plan -
>  > > https://fedoraproject.org/wiki/User:Adamwill/Draft_Fedora_21_test_plan -
>  > > and the discussion of same), but CCing test@ as I know some folks there
>  > > will be interested. Please ensure follow-ups *at least* go to server@
>  > > (no follow-ups only to test@.)
>  > 
>  > I've now cleaned up three of the proposed criteria and posted them in
>  > the 'stock format' here:
>  > 
>  > https://fedoraproject.org/wiki/User:Adamwill/Draft_server_release_criteria
>  
>  I've revised this page again today based on the feedback received so
>  far. Notably, I've tried to make the requirement for both ends of log
>  forwarding to work more clear - I'm still assuming this is actually what
>  we want, if we only want to mandate client not server, someone speak up.
>  I added the remote authentication criterion, with a requirement that it
>  be possible at install time but a stipulation that only being possible
>  non-interactively (via kickstart) is fine. I removed the explicit
>  requirement as regards how conflicts between Role and user-specified
>  firewall config should be resolved. 
Today I've revised the criteria again, adding a set of server role
criteria derived from the tech spec. That URL again:

https://fedoraproject.org/wiki/User:Adamwill/Draft_server_release_criteria

review would be appreciated.

Should we include a criterion requiring remote management of server
roles on a Fedora Server machine to work? How do we envisage this being
configured - at install time? Immediately post-install? From kickstart?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Simo Sorce]

On Fri, 2014-06-06 at 15:55 -0700, Adam Williamson wrote:
>  
>  * It must be possible to join the system to a FreeIPA or Active
>  Directory domain, and the system must respect the identity,
>  authentication and access control configuration provided by the
>  domain.
>  [details as subnotes] 
There is an exception here, and that is the case when the server is
itself a Domain Controller, I guess it should be noted ?

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/11/2014 09:50 PM, Simo Sorce wrote:
>  On Fri, 2014-06-06 at 15:55 -0700, Adam Williamson wrote:
> > 
> > * It must be possible to join the system to a FreeIPA or Active 
> > Directory domain, and the system must respect the identity, 
> > authentication and access control configuration provided by the 
> > domain. [details as subnotes]
>  
>  There is an exception here, and that is the case when the server
>  is itself a Domain Controller, I guess it should be noted ? 
I'd argue that this is just a machine that is joined to a domain that
it just happens to be hosting. So this still meets the criterion.

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlOZu08ACgkQeiVVYja6o6NtfwCgsIopxAG9c9aImb4vUMD+CFHF
> h1UAnRBaQg5KZ4VIkvYNCcqGKPETam4y
> =B+fu
> -----END PGP SIGNATURE-----

[Russell Doty]

What about adding a functional test or smoke test as part of
installation - have an automated test that validates that the server
role is functioning correctly. For example, for a dns server verify that
it can properly resolve addresses.

This would be especially valuable for multi-node applications that
require multiple systems to be properly configured and communicating in
order to work correctly.

This proposed requirement is user centric - it reports whether or not
the application is functioning correctly, rather than a lot of low level
system details.

On Fri, 2014-06-06 at 15:55 -0700, Adam Williamson wrote:
>  Hi, folks. Here's my initial dump of proposed release criteria
> specific
>  to Server. This is primarily a Server WG responsibility (per the draft
>  test plan -
>  https://fedoraproject.org/wiki/User:Adamwill/Draft_Fedora_21_test_plan -
>  and the discussion of same), but CCing test@ as I know some folks there
>  will be interested. Please ensure follow-ups *at least* go to server@
>  (no follow-ups only to test@.)
>  
>  * It must be possible to forward system logs from one system running the
>  release to another using rsyslog.
>  
>  * After system installation, the system firewall must be active, and the
>  only ports which may be open are port 22 and any ports associated with
>  server Roles selected during installation. [pace explicit kickstart
>  configuration]
>  
>  * After system installation, SELinux must be enabled and in enforcing
>  mode. [if user does not explicitly request otherwise via cmdline or
>  kickstart]
>  
>  * It must be possible to join the system to a FreeIPA or Active
>  Directory domain, and the system must respect the identity,
>  authentication and access control configuration provided by the domain.
>  [details as subnotes]
>  
>  The bits in square brackets are rough notes; they indicate things that
>  would be expanded on via the 'expandable sub-notes' mechanism used on
>  the release criteria pages at present (see any of the release criteria
>  pages, e.g.
>  https://fedoraproject.org/wiki/Fedora_21_Alpha_Release_Criteria ). The
>  text given above would be the 'main text' for each criterion. We haven't
>  yet really settled the question of how exactly the presentation of the
>  criteria needs to be done in a Product world, but we can consider the
>  actual additional criteria that are required more or less independently
>  of how they should be presented, I think.
>  
>  This set of criteria are the ones I think need to be added to 'back' the
>  tech spec,
>  https://fedoraproject.org/wiki/Server/Technical_Specification ,
>  *excluding* Role functionality - I expect we'll need a whole subset of
>  criteria for Role functionality, so I figured I can write that up as a
>  separate batch to keep each chunk a manageable size.
>  
>  There are two areas which probably need to be captured with a criterion,
>  but which I couldn't really write one for yet because of vagueness in
>  the spec. One is problem reporting mechanisms - see my separate mail on
>  that topic. The other is remote system management. The spec says:
>  
>  "Software updates on the Fedora Server must be possible to perform
>  either locally using command-line tools (e.g. yum/dnf)..."
>  
>  (okay, we already cover that in current criteria)
>  
>  "...or centrally by common management systems (e.g. Puppet, Chef,
>  Satellite, Spacewalk, OpenLMI)."
>  
>  well, that's extremely broad. Do we really want to have the criteria say
>  "it must be possible to update a Fedora Server system via Puppet, Chef,
>  Satellite, Spacewalk or OpenLMI", write a test case for each, and block
>  releases unless all of those mechanisms work? Or do we want to focus
>  down a bit?
>  
>  Thoughts on the proposed criteria, the remote update issue, or any other
>  criteria we might need beyond the existing 'core' criteria and the ones
>  for Role functionality would be great! Thanks folks.

[Adam Williamson]

On Mon, 2014-06-16 at 11:51 -0400, Russell Doty wrote:
>  What about adding a functional test or smoke test as part of
>  installation - have an automated test that validates that the server
>  role is functioning correctly. For example, for a dns server verify that
>  it can properly resolve addresses.
>  
>  This would be especially valuable for multi-node applications that
>  require multiple systems to be properly configured and communicating in
>  order to work correctly.
>  
>  This proposed requirement is user centric - it reports whether or not
>  the application is functioning correctly, rather than a lot of low level
>  system details. 
This particular thread is about release criteria, not validation tests,
note. It's not comprehensive - note the bit of the post that read
"*excluding* Role functionality - I expect we'll need a whole subset of
criteria for Role functionality, so I figured I can write that up as a
separate batch to keep each chunk a manageable size."

I since drafted up the role criteria and sent an email to server@; see
https://fedoraproject.org/wiki/User:Adamwill/Draft_server_release_criteria and
https://lists.fedoraproject.org/pipermail/server/2014-June/001245.html .

Per-role functional requirements are certainly something we could add,
and probably a good idea, but they'd have to be done as part of actually
creating the roles. We don't have a DNS server role yet, and it's not in
the set of two roles planned for Fedora 21:
https://fedoraproject.org/wiki/Server/Technical_Specification#Supported_R... . Those are
domain controller and database server.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Adam Williamson]

On Mon, 2014-06-16 at 12:39 -0700, Adam Williamson wrote:

>  Per-role functional requirements are certainly something we could
> add,
>  and probably a good idea, but they'd have to be done as part of actually
>  creating the roles. We don't have a DNS server role yet, and it's not in
>  the set of two roles planned for Fedora 21:
>  https://fedoraproject.org/wiki/Server/Technical_Specification#Supported_R... . Those are
> domain controller and database server. 
So thinking about this some more, I think the way it could work is to
have the criteria say something like:

"Primary supported server roles must meet their functional
requirements."

and have it a required part of the server role process that server roles
*have* functional requirements - presumably we'll require some kind of
documentation of server roles, as wiki pages in a standardized
category/naming format or whatever, and the functional requirements can
be a part of that.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Russell Doty]

On Mon, 2014-06-16 at 12:39 -0700, Adam Williamson wrote:
>  On Mon, 2014-06-16 at 11:51 -0400, Russell Doty wrote:
>  > What about adding a functional test or smoke test as part of
>  > installation - have an automated test that validates that the server
>  > role is functioning correctly. For example, for a dns server verify that
>  > it can properly resolve addresses.
>  > 
>  > This would be especially valuable for multi-node applications that
>  > require multiple systems to be properly configured and communicating in
>  > order to work correctly.
>  > 
>  > This proposed requirement is user centric - it reports whether or not
>  > the application is functioning correctly, rather than a lot of low level
>  > system details.
>  
>  This particular thread is about release criteria, not validation tests,
>  note. It's not comprehensive - note the bit of the post that read
>  "*excluding* Role functionality - I expect we'll need a whole subset of
>  criteria for Role functionality, so I figured I can write that up as a
>  separate batch to keep each chunk a manageable size."
>   
Adam, you're absolutely right. I was struck by a thought and posted it
here - but release criteria is not the right place for this discussion.
>  I since drafted up the role criteria and sent an email to server@;
> see
>  https://fedoraproject.org/wiki/User:Adamwill/Draft_server_release_criteria and
> https://lists.fedoraproject.org/pipermail/server/2014-June/001245.html .
>  
>  Per-role functional requirements are certainly something we could add,
>  and probably a good idea, but they'd have to be done as part of actually
>  creating the roles. We don't have a DNS server role yet, and it's not in
>  the set of two roles planned for Fedora 21:
>  https://fedoraproject.org/wiki/Server/Technical_Specification#Supported_R... . Those are
> domain controller and database server.

[Adam Williamson]

On Tue, 2014-06-17 at 10:01 -0400, Russell Doty wrote:

>  > This particular thread is about release criteria, not validation
> tests,
>  > note. It's not comprehensive - note the bit of the post that read
>  > "*excluding* Role functionality - I expect we'll need a whole subset of
>  > criteria for Role functionality, so I figured I can write that up as a
>  > separate batch to keep each chunk a manageable size."
>  > 
>  Adam, you're absolutely right. I was struck by a thought and posted it
>  here - but release criteria is not the right place for this discussion. 
Well, as I posted in a follow-up, once I sat down and thought it
through, I think there *is* room for a criterion there, and people
seemed to generally support the approach in the meeting this morning. So
thanks for the idea!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Russell Doty]

On Tue, 2014-06-17 at 13:31 -0700, Adam Williamson wrote:
>  On Tue, 2014-06-17 at 10:01 -0400, Russell Doty wrote:
>  
>  > > This particular thread is about release criteria, not validation tests,
>  > > note. It's not comprehensive - note the bit of the post that read
>  > > "*excluding* Role functionality - I expect we'll need a whole subset
> of
>  > > criteria for Role functionality, so I figured I can write that up as a
>  > > separate batch to keep each chunk a manageable size."
>  > > 
>  > Adam, you're absolutely right. I was struck by a thought and posted it
>  > here - but release criteria is not the right place for this discussion.
>  
>  Well, as I posted in a follow-up, once I sat down and thought it
>  through, I think there *is* room for a criterion there, and people
>  seemed to generally support the approach in the meeting this morning. So
>  thanks for the idea! 
Well, in that case, "how may I help you sir?"

Initial set of proposed release criteria for Server product - created accounts

[Russell Doty]

Do we want to say anything about accounts created during application
installation?

Such as:

* Any accounts created for an application or service must be configured
to disallow interactive login.

* Do we need any restrictions on privileged accounts?

* Any other guidelines for accounts?

On Fri, 2014-06-06 at 15:55 -0700, Adam Williamson wrote:
>  Hi, folks. Here's my initial dump of proposed release criteria
> specific
>  to Server. This is primarily a Server WG responsibility (per the draft
>  test plan -
>  https://fedoraproject.org/wiki/User:Adamwill/Draft_Fedora_21_test_plan -
>  and the discussion of same), but CCing test@ as I know some folks there
>  will be interested. Please ensure follow-ups *at least* go to server@
>  (no follow-ups only to test@.)
>  
>  * It must be possible to forward system logs from one system running the
>  release to another using rsyslog.
>  
>  * After system installation, the system firewall must be active, and the
>  only ports which may be open are port 22 and any ports associated with
>  server Roles selected during installation. [pace explicit kickstart
>  configuration]
>  
>  * After system installation, SELinux must be enabled and in enforcing
>  mode. [if user does not explicitly request otherwise via cmdline or
>  kickstart]
>  
>  * It must be possible to join the system to a FreeIPA or Active
>  Directory domain, and the system must respect the identity,
>  authentication and access control configuration provided by the domain.
>  [details as subnotes]
>  
>  The bits in square brackets are rough notes; they indicate things that
>  would be expanded on via the 'expandable sub-notes' mechanism used on
>  the release criteria pages at present (see any of the release criteria
>  pages, e.g.
>  https://fedoraproject.org/wiki/Fedora_21_Alpha_Release_Criteria ). The
>  text given above would be the 'main text' for each criterion. We haven't
>  yet really settled the question of how exactly the presentation of the
>  criteria needs to be done in a Product world, but we can consider the
>  actual additional criteria that are required more or less independently
>  of how they should be presented, I think.
>  
>  This set of criteria are the ones I think need to be added to 'back' the
>  tech spec,
>  https://fedoraproject.org/wiki/Server/Technical_Specification ,
>  *excluding* Role functionality - I expect we'll need a whole subset of
>  criteria for Role functionality, so I figured I can write that up as a
>  separate batch to keep each chunk a manageable size.
>  
>  There are two areas which probably need to be captured with a criterion,
>  but which I couldn't really write one for yet because of vagueness in
>  the spec. One is problem reporting mechanisms - see my separate mail on
>  that topic. The other is remote system management. The spec says:
>  
>  "Software updates on the Fedora Server must be possible to perform
>  either locally using command-line tools (e.g. yum/dnf)..."
>  
>  (okay, we already cover that in current criteria)
>  
>  "...or centrally by common management systems (e.g. Puppet, Chef,
>  Satellite, Spacewalk, OpenLMI)."
>  
>  well, that's extremely broad. Do we really want to have the criteria say
>  "it must be possible to update a Fedora Server system via Puppet, Chef,
>  Satellite, Spacewalk or OpenLMI", write a test case for each, and block
>  releases unless all of those mechanisms work? Or do we want to focus
>  down a bit?
>  
>  Thoughts on the proposed criteria, the remote update issue, or any other
>  criteria we might need beyond the existing 'core' criteria and the ones
>  for Role functionality would be great! Thanks folks.

Initial set of proposed release criteria for Server product - created accounts

[Adam Williamson]

On Mon, 2014-06-16 at 12:47 -0400, Russell Doty wrote:
>  Do we want to say anything about accounts created during application
>  installation?
>  
>  Such as:
>  
>  * Any accounts created for an application or service must be configured
>  to disallow interactive login.
>  
>  * Do we need any restrictions on privileged accounts?
>  
>  * Any other guidelines for accounts? 
At least to me these seem more like packaging / role creation guidelines
than release criteria, though it's kind of a fuzzy thing.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

Initial set of proposed release criteria for Server product - Resource Limitations

[Russell Doty]

Do we want applications to perform resource management during
installation and configuration?

For example:

* Specify maximum CPU utilization
* Specify maximum memory utilization
* Specify maximum network bandwidth
* If threaded, specify maximum number of threads
* Specify maximum number of file handles

Any other resource contention or exhaustion cases we might want to deal
with?

On Fri, 2014-06-06 at 15:55 -0700, Adam Williamson wrote:
>  Hi, folks. Here's my initial dump of proposed release criteria
> specific
>  to Server. This is primarily a Server WG responsibility (per the draft
>  test plan -
>  https://fedoraproject.org/wiki/User:Adamwill/Draft_Fedora_21_test_plan -
>  and the discussion of same), but CCing test@ as I know some folks there
>  will be interested. Please ensure follow-ups *at least* go to server@
>  (no follow-ups only to test@.)
>  
>  * It must be possible to forward system logs from one system running the
>  release to another using rsyslog.
>  
>  * After system installation, the system firewall must be active, and the
>  only ports which may be open are port 22 and any ports associated with
>  server Roles selected during installation. [pace explicit kickstart
>  configuration]
>  
>  * After system installation, SELinux must be enabled and in enforcing
>  mode. [if user does not explicitly request otherwise via cmdline or
>  kickstart]
>  
>  * It must be possible to join the system to a FreeIPA or Active
>  Directory domain, and the system must respect the identity,
>  authentication and access control configuration provided by the domain.
>  [details as subnotes]
>  
>  The bits in square brackets are rough notes; they indicate things that
>  would be expanded on via the 'expandable sub-notes' mechanism used on
>  the release criteria pages at present (see any of the release criteria
>  pages, e.g.
>  https://fedoraproject.org/wiki/Fedora_21_Alpha_Release_Criteria ). The
>  text given above would be the 'main text' for each criterion. We haven't
>  yet really settled the question of how exactly the presentation of the
>  criteria needs to be done in a Product world, but we can consider the
>  actual additional criteria that are required more or less independently
>  of how they should be presented, I think.
>  
>  This set of criteria are the ones I think need to be added to 'back' the
>  tech spec,
>  https://fedoraproject.org/wiki/Server/Technical_Specification ,
>  *excluding* Role functionality - I expect we'll need a whole subset of
>  criteria for Role functionality, so I figured I can write that up as a
>  separate batch to keep each chunk a manageable size.
>  
>  There are two areas which probably need to be captured with a criterion,
>  but which I couldn't really write one for yet because of vagueness in
>  the spec. One is problem reporting mechanisms - see my separate mail on
>  that topic. The other is remote system management. The spec says:
>  
>  "Software updates on the Fedora Server must be possible to perform
>  either locally using command-line tools (e.g. yum/dnf)..."
>  
>  (okay, we already cover that in current criteria)
>  
>  "...or centrally by common management systems (e.g. Puppet, Chef,
>  Satellite, Spacewalk, OpenLMI)."
>  
>  well, that's extremely broad. Do we really want to have the criteria say
>  "it must be possible to update a Fedora Server system via Puppet, Chef,
>  Satellite, Spacewalk or OpenLMI", write a test case for each, and block
>  releases unless all of those mechanisms work? Or do we want to focus
>  down a bit?
>  
>  Thoughts on the proposed criteria, the remote update issue, or any other
>  criteria we might need beyond the existing 'core' criteria and the ones
>  for Role functionality would be great! Thanks folks.

Initial set of proposed release criteria for Server product - Resource Limitations

[Adam Williamson]

On Mon, 2014-06-16 at 12:51 -0400, Russell Doty wrote:
>  Do we want applications to perform resource management during
>  installation and configuration?
>  
>  For example:
>  
>  * Specify maximum CPU utilization
>  * Specify maximum memory utilization
>  * Specify maximum network bandwidth
>  * If threaded, specify maximum number of threads
>  * Specify maximum number of file handles
>  
>  Any other resource contention or exhaustion cases we might want to deal
>  with? 
That goes rather beyond the current technical specification. It's not a
good idea to do product design via release criteria; it's rather going
the wrong way around. It'd be best to propose these in a separate thread
as potential new aspects of Server's technical design...
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net