[F24 Criteria Change] Cockpit Release Criteria

Máirín Duffy duffy at redhat.com
Thu Nov 5 16:10:30 UTC 2015


Hi!

On 11/02/2015 08:31 AM, Stephen Gallagher wrote:
> * All Cockpit functional criteria must be satisfied when the user is
> running any of the following blocking browsers:
>   - Mozilla Firefox as shipped in the same Fedora release
>   - At least one of a) Mozilla Firefox or b) Google Chrome of the latest
>     available version on Windows at compose time.
>   - At least one of a) Mozilla Firefox or b) Google Chrome of the latest
>     available version on OSX at compose time.

So we talked about this at the meeting yesterday [1] and I committed to 
writing a draft that reflected what came out of that discussion.

Everything below relies on the assumption that any given issue found is 
an issue in Fedora / Cockpit, not in the browser/platform experiencing 
the issue.

So first I'll put the draft, a modified version of Stephen's, then the 
discussion / rationale behind it. Let me know what you think?

Draft
=====

* All Cockpit functional criteria must be satisfied when the user is 
running any of the following platforms:

- Mozilla Firefox as shipped in the same Fedora release
- Google Chrome of the latest available version at compose time on the 
same Fedora release.
- At least one of a) Mozilla Firefox or b) Google Chrome of the latest
available version on Windows at compose time.
- At least one of a) Mozilla Firefox or b) Google Chrome of the latest
available version on OSX at compose time.

Manual testing will occur on the Firefox/Fedora platform. It will not be 
performed on the Chrome/Fedora platform nor any of the Windows / OSX 
platforms mentioned above, but if issues in meeting the functional 
criteria are reported against those platforms, they may block the release.


Discussion
==========

What Platform(s) Experience the Issue?
--------------------------------------

So basically we talked about how there's some different axes around 
browser support for a web app:

1) What specific browser is affected? Which ones do we care about?

2) Is it on a desktop/laptop vs mobile device (tablet / phone)? Which do 
we care about?

3) Is the OS Fedora, a Linux, OS X, Windows? Which do we care about?


Does It Block or Not?
---------------------

Then, at what level do we support whatever combinations we think are 
important?

- Tier 1: Manual testing on Fedora, issues found could block release

- Tier 2: Not manually tested, but we care about them so if somebody 
finds an issue it could block release

- Tier 3: Not manually tested, not a critical platform to us so we won't 
block if an issue is found & reported.


Platform Axis: Browsers
-----------------------

First, on browsers - according to the W3C's w3schools site, they see 
about 60-65% of visitors on Chrome and Firefox around 20-24%. IE is 3rd 
at around 6-8%. Other browser usage statistics reports seemed to have 
similar numbers.

Since they are the top two (and Firefox is also our default workstation 
browser), it seems to make sense to put both Chrome and Firefox into Tier 1.


Platform Axis: Laptop/desktop vs Mobile
---------------------------------------

We talked about laptop/desktop usage being more critical than mobile 
(although Cockpit does care and think about mobile usage.)


Platform Axis: OS
-----------------

We talked about concerns about forcing people to have to test on an OS 
they weren't comfortable running. However, we are trying to expand our 
userbase which means being friendly to folks who run the most popular 
OSes - Windows and OS X. We wouldn't do manual testing of Windows or OS 
X, but we might care if an issue was submitted to us that made Cockpit 
unable to meet its functional criteria on that OS. Adam clarified (if I 
understand correctly) that having this as a criteria doesn't dictate 
manual testing.

Summary
-------

With respect to tiers:

Tier 1:
Firefox on Fedora

Tier 2:
Chrome on Fedora
Firefox on Windows
Chrome on Windows
Firefox on OS X
Chrome on OS X

Tier 3:
Anything else.


~m


[1] 
https://meetbot.fedoraproject.org/fedora-meeting-1/2015-11-03/fedora-meeting-1.2015-11-03-16.00.html

[2] http://www.w3schools.com/browsers/browsers_stats.asp


More information about the server mailing list