-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Recently, we discovered a bug in gnutls that caused Cockpit to be unreachable by recent versions of Google Chrome. It was ambiguous what the release criteria actually means, since it didn't specify which browser applications were blocking. I'd like to propose the following additional wording for Cockpit criteria:
* All Cockpit functional criteria must be satisfied when the user is running any of the following blocking browsers: - Mozilla Firefox as shipped in the same Fedora release - Mozilla Firefox of the latest available version on Windows at compose time. - Mozilla Firefox of the latest available version on OSX at compose time. - Google Chrome of the latest available version on Fedora at compose time. - Google Chrome of the latest available version on Windows at compose time. - Google Chrome of the latest available version on OSX at compose time.
Alternately, we could decide that it's only *blocking* if the above browsers work with Cockpit when the browser is running on Fedora, but that is somewhat at odds with our reasoning for having a management console as a web UI in the first place: that it is accessible regardless of the client system.
Comments welcome, but please keep replies on the test@lists.fedoraproject.org list, as that's where criteria decisions are made.
On 22.10.2015 20:35, Stephen Gallagher wrote:
Recently, we discovered a bug in gnutls that caused Cockpit to be unreachable by recent versions of Google Chrome. It was ambiguous what the release criteria actually means, since it didn't specify which browser applications were blocking. I'd like to propose the following additional wording for Cockpit criteria:
- All Cockpit functional criteria must be satisfied when the user is
running any of the following blocking browsers:
- Mozilla Firefox as shipped in the same Fedora release
- Mozilla Firefox of the latest available version on Windows at compose time.
- Mozilla Firefox of the latest available version on OSX at compose time.
- Google Chrome of the latest available version on Fedora at compose time.
- Google Chrome of the latest available version on Windows at compose time.
- Google Chrome of the latest available version on OSX at compose time.
I'm in favor of this.
But if we're going to make this a requirement, then we should put some time into getting Selenium working with the above, and checking these as part of our routine testing.
Stef
On 22/10/15 14:35, Stephen Gallagher wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Recently, we discovered a bug in gnutls that caused Cockpit to be unreachable by recent versions of Google Chrome. It was ambiguous what the release criteria actually means, since it didn't specify which browser applications were blocking. I'd like to propose the following additional wording for Cockpit criteria:
- All Cockpit functional criteria must be satisfied when the user is
running any of the following blocking browsers:
- Mozilla Firefox as shipped in the same Fedora release
- Mozilla Firefox of the latest available version on Windows at compose time.
- Mozilla Firefox of the latest available version on OSX at compose time.
- Google Chrome of the latest available version on Fedora at compose time.
- Google Chrome of the latest available version on Windows at compose time.
- Google Chrome of the latest available version on OSX at compose time.
Alternately, we could decide that it's only *blocking* if the above browsers work with Cockpit when the browser is running on Fedora, but that is somewhat at odds with our reasoning for having a management console as a web UI in the first place: that it is accessible regardless of the client system.
Comments welcome, but please keep replies on the test@lists.fedoraproject.org list, as that's where criteria decisions are made.
although I totally understand the reason I am against a release criteria *for Fedora* that forces a tester to install non-free Software (both OS *and* Browser).
Simo.
On 10/23/2015 04:42 PM, Simo Sorce wrote:
On 22/10/15 14:35, Stephen Gallagher wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Recently, we discovered a bug in gnutls that caused Cockpit to be unreachable by recent versions of Google Chrome. It was ambiguous what the release criteria actually means, since it didn't specify which browser applications were blocking. I'd like to propose the following additional wording for Cockpit criteria:
- All Cockpit functional criteria must be satisfied when the user is
running any of the following blocking browsers:
- Mozilla Firefox as shipped in the same Fedora release
- Mozilla Firefox of the latest available version on Windows at compose time.
- Mozilla Firefox of the latest available version on OSX at compose time.
- Google Chrome of the latest available version on Fedora at compose time.
- Google Chrome of the latest available version on Windows at compose time.
- Google Chrome of the latest available version on OSX at compose time.
Alternately, we could decide that it's only *blocking* if the above browsers work with Cockpit when the browser is running on Fedora, but that is somewhat at odds with our reasoning for having a management console as a web UI in the first place: that it is accessible regardless of the client system.
Comments welcome, but please keep replies on the test@lists.fedoraproject.org list, as that's where criteria decisions are made.
although I totally understand the reason I am against a release criteria *for Fedora* that forces a tester to install non-free Software (both OS *and* Browser).
Simo.
I'm with Simo. I *DO NOT* want to place a burden on a FOSS operating system to be 100% compatible with non-FOSS software.
On Wed, Oct 28, 2015 at 5:11 PM, Dan Mossor danofsatx@gmail.com wrote:
I'm with Simo. I *DO NOT* want to place a burden on a FOSS operating system to be 100% compatible with non-FOSS software.
OK well seeing as we even have some issues with FOSS software A not getting along with FOSS software B and there's no burden...
The question is: should Cockpit have a bug that prevents it from being functional with OSX+Chrome, should it be blocking? What if it's OSX+Firefox?
I would say a more practical, and permissive policy would be: should Cockpit have a bug that prevents it from being functional with OSX or Windows using any (Chrome, FireFox, or OS default) then it's a blocking bug. The point is to make sure a Cockpit bug doesn't prevent people on other OS's from using Cockpit and thus benefiting.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/01/2015 04:23 AM, Chris Murphy wrote:
On Wed, Oct 28, 2015 at 5:11 PM, Dan Mossor danofsatx@gmail.com wrote:
I'm with Simo. I *DO NOT* want to place a burden on a FOSS operating system to be 100% compatible with non-FOSS software.
OK well seeing as we even have some issues with FOSS software A not getting along with FOSS software B and there's no burden...
The question is: should Cockpit have a bug that prevents it from being functional with OSX+Chrome, should it be blocking? What if it's OSX+Firefox?
I would say a more practical, and permissive policy would be: should Cockpit have a bug that prevents it from being functional with OSX or Windows using any (Chrome, FireFox, or OS default) then it's a blocking bug. The point is to make sure a Cockpit bug doesn't prevent people on other OS's from using Cockpit and thus benefiting.
That's a fairly reasonable statement. So something like:
* All Cockpit functional criteria must be satisfied when the user is running any of the following blocking browsers: - Mozilla Firefox as shipped in the same Fedora release - At least one of a) Mozilla Firefox or b) Google Chrome of the latest available version on Windows at compose time. - At least one of a) Mozilla Firefox or b) Google Chrome of the latest available version on OSX at compose time.
Would that be satisfactory to everyone?
On 02/11/15 08:31, Stephen Gallagher wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/01/2015 04:23 AM, Chris Murphy wrote:
On Wed, Oct 28, 2015 at 5:11 PM, Dan Mossor danofsatx@gmail.com wrote:
I'm with Simo. I *DO NOT* want to place a burden on a FOSS operating system to be 100% compatible with non-FOSS software.
OK well seeing as we even have some issues with FOSS software A not getting along with FOSS software B and there's no burden...
The question is: should Cockpit have a bug that prevents it from being functional with OSX+Chrome, should it be blocking? What if it's OSX+Firefox?
I would say a more practical, and permissive policy would be: should Cockpit have a bug that prevents it from being functional with OSX or Windows using any (Chrome, FireFox, or OS default) then it's a blocking bug. The point is to make sure a Cockpit bug doesn't prevent people on other OS's from using Cockpit and thus benefiting.
That's a fairly reasonable statement. So something like:
- All Cockpit functional criteria must be satisfied when the user is
running any of the following blocking browsers:
- Mozilla Firefox as shipped in the same Fedora release
- At least one of a) Mozilla Firefox or b) Google Chrome of the latest available version on Windows at compose time.
- At least one of a) Mozilla Firefox or b) Google Chrome of the latest available version on OSX at compose time.
Would that be satisfactory to everyone?
As long as this does not mean people have to go out and find those platforms for testing.
I think it is reasonable to state that there shouldn't be known blocker bugs for those platforms, but the onus of testing is on those that like them and check on their own. A release criteria must not force anyone to test proprietary platforms if there are no know bugs for them, and a bug resolution from upstream should be sufficient for us to call it done without required testing by Fedora QA or other volunteers.
Simo.
Hi!
On 11/02/2015 08:31 AM, Stephen Gallagher wrote:
- All Cockpit functional criteria must be satisfied when the user is
running any of the following blocking browsers:
- Mozilla Firefox as shipped in the same Fedora release
- At least one of a) Mozilla Firefox or b) Google Chrome of the latest available version on Windows at compose time.
- At least one of a) Mozilla Firefox or b) Google Chrome of the latest available version on OSX at compose time.
So we talked about this at the meeting yesterday [1] and I committed to writing a draft that reflected what came out of that discussion.
Everything below relies on the assumption that any given issue found is an issue in Fedora / Cockpit, not in the browser/platform experiencing the issue.
So first I'll put the draft, a modified version of Stephen's, then the discussion / rationale behind it. Let me know what you think?
Draft =====
* All Cockpit functional criteria must be satisfied when the user is running any of the following platforms:
- Mozilla Firefox as shipped in the same Fedora release - Google Chrome of the latest available version at compose time on the same Fedora release. - At least one of a) Mozilla Firefox or b) Google Chrome of the latest available version on Windows at compose time. - At least one of a) Mozilla Firefox or b) Google Chrome of the latest available version on OSX at compose time.
Manual testing will occur on the Firefox/Fedora platform. It will not be performed on the Chrome/Fedora platform nor any of the Windows / OSX platforms mentioned above, but if issues in meeting the functional criteria are reported against those platforms, they may block the release.
Discussion ==========
What Platform(s) Experience the Issue? --------------------------------------
So basically we talked about how there's some different axes around browser support for a web app:
1) What specific browser is affected? Which ones do we care about?
2) Is it on a desktop/laptop vs mobile device (tablet / phone)? Which do we care about?
3) Is the OS Fedora, a Linux, OS X, Windows? Which do we care about?
Does It Block or Not? ---------------------
Then, at what level do we support whatever combinations we think are important?
- Tier 1: Manual testing on Fedora, issues found could block release
- Tier 2: Not manually tested, but we care about them so if somebody finds an issue it could block release
- Tier 3: Not manually tested, not a critical platform to us so we won't block if an issue is found & reported.
Platform Axis: Browsers -----------------------
First, on browsers - according to the W3C's w3schools site, they see about 60-65% of visitors on Chrome and Firefox around 20-24%. IE is 3rd at around 6-8%. Other browser usage statistics reports seemed to have similar numbers.
Since they are the top two (and Firefox is also our default workstation browser), it seems to make sense to put both Chrome and Firefox into Tier 1.
Platform Axis: Laptop/desktop vs Mobile ---------------------------------------
We talked about laptop/desktop usage being more critical than mobile (although Cockpit does care and think about mobile usage.)
Platform Axis: OS -----------------
We talked about concerns about forcing people to have to test on an OS they weren't comfortable running. However, we are trying to expand our userbase which means being friendly to folks who run the most popular OSes - Windows and OS X. We wouldn't do manual testing of Windows or OS X, but we might care if an issue was submitted to us that made Cockpit unable to meet its functional criteria on that OS. Adam clarified (if I understand correctly) that having this as a criteria doesn't dictate manual testing.
Summary -------
With respect to tiers:
Tier 1: Firefox on Fedora
Tier 2: Chrome on Fedora Firefox on Windows Chrome on Windows Firefox on OS X Chrome on OS X
Tier 3: Anything else.
~m
[1] https://meetbot.fedoraproject.org/fedora-meeting-1/2015-11-03/fedora-meeting...
Máirín Duffy duffy@redhat.com writes:
So first I'll put the draft, a modified version of Stephen's, then the discussion / rationale behind it. Let me know what you think?
This is excellent!
On Nov 5, 2015, at 5:10 PM, Máirín Duffy duffy@redhat.com wrote:
Hi!
On 11/02/2015 08:31 AM, Stephen Gallagher wrote:
- All Cockpit functional criteria must be satisfied when the user is
running any of the following blocking browsers:
- Mozilla Firefox as shipped in the same Fedora release
- At least one of a) Mozilla Firefox or b) Google Chrome of the latest available version on Windows at compose time.
- At least one of a) Mozilla Firefox or b) Google Chrome of the latest available version on OSX at compose time.
So we talked about this at the meeting yesterday [1] and I committed to writing a draft that reflected what came out of that discussion.
Everything below relies on the assumption that any given issue found is an issue in Fedora / Cockpit, not in the browser/platform experiencing the issue.
So first I'll put the draft, a modified version of Stephen's, then the discussion / rationale behind it. Let me know what you think?
Draft
- All Cockpit functional criteria must be satisfied when the user is running any of the following platforms:
- Mozilla Firefox as shipped in the same Fedora release
- Google Chrome of the latest available version at compose time on the same Fedora release.
- At least one of a) Mozilla Firefox or b) Google Chrome of the latest
available version on Windows at compose time.
- At least one of a) Mozilla Firefox or b) Google Chrome of the latest
available version on OSX at compose time.
Manual testing will occur on the Firefox/Fedora platform. It will not be performed on the Chrome/Fedora platform nor any of the Windows / OSX platforms mentioned above, but if issues in meeting the functional criteria are reported against those platforms, they may block the release.
I think this is basically perfect. Thank you very much for putting this together, particularly the detailed rationale.
The only thing I might change is "Manual testing will occur on the Firefox/Fedora platform" to "Mandatory testing will occur..."
I for one keep several systems around for cross-compatibility testing and will most likely do regular spot-checks that things aren't completely broken on the non-Fedora platforms. I note this because the practical effect here will be that any of these combinations here may indeed block the release (but I'm perfectly happy not requiring anyone else to use those other systems for testing).
Discussion
What Platform(s) Experience the Issue?
So basically we talked about how there's some different axes around browser support for a web app:
What specific browser is affected? Which ones do we care about?
Is it on a desktop/laptop vs mobile device (tablet / phone)? Which do we care about?
Is the OS Fedora, a Linux, OS X, Windows? Which do we care about?
Does It Block or Not?
Then, at what level do we support whatever combinations we think are important?
Tier 1: Manual testing on Fedora, issues found could block release
Tier 2: Not manually tested, but we care about them so if somebody finds an issue it could block release
Tier 3: Not manually tested, not a critical platform to us so we won't block if an issue is found & reported.
Platform Axis: Browsers
First, on browsers - according to the W3C's w3schools site, they see about 60-65% of visitors on Chrome and Firefox around 20-24%. IE is 3rd at around 6-8%. Other browser usage statistics reports seemed to have similar numbers.
Since they are the top two (and Firefox is also our default workstation browser), it seems to make sense to put both Chrome and Firefox into Tier 1.
Platform Axis: Laptop/desktop vs Mobile
We talked about laptop/desktop usage being more critical than mobile (although Cockpit does care and think about mobile usage.)
Platform Axis: OS
We talked about concerns about forcing people to have to test on an OS they weren't comfortable running. However, we are trying to expand our userbase which means being friendly to folks who run the most popular OSes - Windows and OS X. We wouldn't do manual testing of Windows or OS X, but we might care if an issue was submitted to us that made Cockpit unable to meet its functional criteria on that OS. Adam clarified (if I understand correctly) that having this as a criteria doesn't dictate manual testing.
Summary
With respect to tiers:
Tier 1: Firefox on Fedora
Tier 2: Chrome on Fedora Firefox on Windows Chrome on Windows Firefox on OS X Chrome on OS X
Tier 3: Anything else.
~m
[1] https://meetbot.fedoraproject.org/fedora-meeting-1/2015-11-03/fedora-meeting...
[2] http://www.w3schools.com/browsers/browsers_stats.asp _______________________________________________ server mailing list server@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/server
On 11/05/2015 10:10 AM, Máirín Duffy wrote:
So we talked about this at the meeting yesterday [1] and I committed to writing a draft that reflected what came out of that discussion.
Everything below relies on the assumption that any given issue found is an issue in Fedora / Cockpit, not in the browser/platform experiencing the issue.
This is fantastic and does a good job of giving a step-by-step methodology for testing the front-end (which is exactly what we need).
Nice job!
-- Major Hayden
On Thu, Nov 05, 2015 at 11:10:30AM -0500, Máirín Duffy wrote:
First, on browsers - according to the W3C's w3schools site, they see about 60-65% of visitors on Chrome and Firefox around 20-24%. IE is 3rd at around 6-8%. Other browser usage statistics reports seemed to have similar numbers.
Note that W3C is not affiliated with w3schools in any way.
On 11/06/2015 12:54 PM, Lars Seipel wrote:
On Thu, Nov 05, 2015 at 11:10:30AM -0500, Máirín Duffy wrote:
First, on browsers - according to the W3C's w3schools site, they see about 60-65% of visitors on Chrome and Firefox around 20-24%. IE is 3rd at around 6-8%. Other browser usage statistics reports seemed to have similar numbers.
Note that W3C is not affiliated with w3schools in any way.
Gah, my bad. A w3c person originally referred me to it years and years ago and I just had that connection in my head. o_O
~m
Hi folks,
Just a quick follow-up on this -
On 11/05/2015 11:10 AM, Máirín Duffy wrote:
So we talked about this at the meeting yesterday [1] and I committed to writing a draft that reflected what came out of that discussion.
So we voted on this in the meeting today; with some slight language additions (to point out only issues we have control of can be blockers; workarounds for issues in 3rd party code can be freeze exceptions only.)
I added it to the final release criteria for server in f24 here:
https://fedoraproject.org/wiki/Fedora_24_Final_Release_Criteria#Server_Produ...
~m
server@lists.fedoraproject.org
-
Chris Murphy -
Dan Mossor -
Lars Seipel -
Major Hayden -
Marius Vollmer -
Máirín Duffy -
Simo Sorce -
Stef Walter -
Stephen Gallagher