[F24 Criteria Change] Cockpit Release Criteria

[Adam Williamson]

On Fri, 2015-10-23 at 07:41 -0500, Dennis Gilmore wrote:
> On Thursday, October 22, 2015 02:35:11 PM Stephen Gallagher wrote:
> > Recently, we discovered a bug in gnutls that caused Cockpit to be
> > unreachable by recent versions of Google Chrome. It was ambiguous
> > what
> > the release criteria actually means, since it didn't specify which
> > browser applications were blocking. I'd like to propose the
> > following
> > additional wording for Cockpit criteria:
> > 
> > * All Cockpit functional criteria must be satisfied when the user
> > is
> > running any of the following blocking browsers:
> >  - Mozilla Firefox as shipped in the same Fedora release
> >  - Mozilla Firefox of the latest available version on Windows at
> >    compose time.
> >  - Mozilla Firefox of the latest available version on OSX at
> > compose
> >    time.
> >  - Google Chrome of the latest available version on Fedora at
> > compose
> >    time.
> >  - Google Chrome of the latest available version on Windows at
> > compose
> >    time.
> >  - Google Chrome of the latest available version on OSX at compose
> > time.
> > 
> > 
> > Alternately, we could decide that it's only *blocking* if the above
> > browsers work with Cockpit when the browser is running on Fedora,
> > but
> > that is somewhat at odds with our reasoning for having a management
> > console as a web UI in the first place: that it is accessible
> > regardless of the client system.
> 
> I think that it is fine. But you need to make sure you have resources
> available to test on Windows and OS X.

Not necessarily. we don't have an exact 1:1 mapping of testing to
criteria. We test some stuff beyond the criteria, and we don't test
some stuff that *is* in the criteria; things like the 'data corruption'
criterion aren't realistically testable, exactly.

There's a precedent that it's OK to have criteria which basically work
this way: if someone reports a violation and it gets nominated, it will
be approved.

>  I wonder what can be done to do 
> automated testing on the platforms to ensure things work.

There are various 'see how your site looks in X' services, I don't know
much about any of them though.

>  I would like to have 
> us try and automate most if not all of the validation,  at least in a
> basic 
> level.

This is exactly what we've been doing with openQA for the last twelve
months.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net