5:20 p.m.
This proposal intends to reduce the scope of the “*Default application
functionality*” release criterion [0]:
All applications that can be launched using the standard graphical
mechanism of a release-blocking desktop after a default installation
of
that desktop must start successfully and withstand a basic functionality
test.
*= Background =*
The area which QA is responsible for has been growing and growing in recent
years. We used to have just a single Fedora release with two blocking
desktops (GNOME and KDE). Then Editions got introduced and we started
testing Workstation, Server, Cloud and the KDE Spin. Additional
architectures were introduced (fortunately i386 got obsolete) and we
started testing and blocking on specific images on armhfp and aarch64.
Currently there are 13 release blocking deliverables mentioned on the
ReleaseBlocking page [1]. That list is not complete, though. Fedora CoreOS
became an official edition lately, and even though its release cycle is not
tied to traditional Fedora release cycle (and that’s why it’s not mentioned
on that wiki page), as an official edition QA should care about it as well.
It is the same story with Fedora IoT, another recent release-blocking
addition [2]. Desktop testing is one of the most time-consuming jobs with
the most frequent bug occurrence. Right now, we’re supposed to be fully
testing and blocking on GNOME, KDE and XFCE (although XFCE might get
dropped in favor or aarch64 Workstation [3]). We cannot test all of this
and honestly claim that we verified basic functionality of all the included
apps on all these desktops. I believe we need to adjust the criterion and
align it closer with reality. In my eyes, it’s better to have a narrower
scope and perform it well than having a large scope and perform it poorly.
*= Proposal =*
Change the criterion to something along these lines:
All applications that can be launched using the standard graphical
mechanism after a default installation of Fedora Workstation on
x86_64
architecture must start successfully and withstand a basic functionality
test.
For other release-blocking desktops (on any architecture), the
requirements only apply to the following types of applications:
* web browser (e.g. firefox) [4]
* file browser (e.g. nautilus)
* package manager (e.g. gnome-software)
* image viewer (e.g. eog)
* document viewer (e.g. evince)
* text editor (e.g. gedit)
* archive manager (e.g. file-roller)
* terminal emulator (e.g. gnome-terminal) [4]
* problem reporter (e.g. abrt)
If there are multiple applications of the same type (e.g. several web
browsers), only one of them needs to satisfy the requirements.
As you can see, the original criterion was kept for Fedora’s flagship
desktop edition, the one that is most prominent on https://getfedora.org
and probably the one that most newcomers download. We would still verify
everything on Fedora Workstation on x86_64. But any other desktop
(including Workstation on aarch64) would get just reduced criteria, because
we simply can’t ensure the same quality bar for the smaller desktop
editions/spins. There are some high-profile types of applications that I
considered including in the list above, but didn’t in the end:
* word editor (e.g. libreoffice-writer)
* spreadsheet editor (e.g. libreoffice-calc)
* video player (e.g. totem)
* help viewer (e.g. yelp)
I’d like to hear your thoughts on whether they should be included or not.
Of course from an end-user point of view, it would be beneficial. But the
question is whether we as QA can promise their testing. And also whether we
want to block the release e.g. if Gnumeric is broken on armhfp XFCE or if
totem doesn’t work on aarch64. Yes, it’s unpleasant, but people using
alternative desktops and architectures are usually far from beginners. It’s
usually not difficult to install a different application. Also note that
the apps included in x86_64 Workstation will be thoroughly tested so they
should be very likely to work well even on other architectures (minus some
arch-specific issues).
I’d like to target Fedora 32 with this proposal, which means we should
decide on this proposal fast. (I wanted to propose it much sooner, but I
was waiting on clarifications about new release-blocking images and also on
some fesco tickets [3], some of which is still not clarified; so I’m sorry
about a late proposal).
Please comment, thanks.
[0]
https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Default_a...
[1] https://fedoraproject.org/wiki/Releases/32/ReleaseBlocking
[2] https://pagure.io/fedora-iot/issue/23
[3] https://pagure.io/fesco/issue/2339
[4] These two are required to work even in Basic release criteria [5], but
I decided to include them anyway to avoid confusion (“why is the web
browser missing?”), and to make it clear that they need to satisfy the
basic functionality (whereas for Basic release criteria just the barebone
functionality might be deemed enough)
[5]
https://fedoraproject.org/wiki/Basic_Release_Criteria#Required_applications