Yesterday I sent out a QA-related proposal that directly affects the release criteria for KDE x86_64 and Workstation x86_64 and (probably) aarch64 images. The proposal is attached below and also available in the test list (including some follow-ups which are recommended to read): https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/t...
Can you please review that proposal and post your thoughts, if you have any? (Responding to test list is preferred, so that we can discuss it in a single place, but responding here is fine as well). Thank you.
---------- Forwarded message --------- From: Kamil Paral kparal@redhat.com Date: Mon, Feb 17, 2020 at 3:20 PM Subject: proposal: Default application functionality criterion reduction To: test test@lists.fedoraproject.org
This proposal intends to reduce the scope of the “*Default application functionality*” release criterion [0]:
All applications that can be launched using the standard graphical
mechanism of a release-blocking desktop after a default installation of that desktop must start successfully and withstand a basic functionality test.
*= Background =*
The area which QA is responsible for has been growing and growing in recent years. We used to have just a single Fedora release with two blocking desktops (GNOME and KDE). Then Editions got introduced and we started testing Workstation, Server, Cloud and the KDE Spin. Additional architectures were introduced (fortunately i386 got obsolete) and we started testing and blocking on specific images on armhfp and aarch64. Currently there are 13 release blocking deliverables mentioned on the ReleaseBlocking page [1]. That list is not complete, though. Fedora CoreOS became an official edition lately, and even though its release cycle is not tied to traditional Fedora release cycle (and that’s why it’s not mentioned on that wiki page), as an official edition QA should care about it as well. It is the same story with Fedora IoT, another recent release-blocking addition [2]. Desktop testing is one of the most time-consuming jobs with the most frequent bug occurrence. Right now, we’re supposed to be fully testing and blocking on GNOME, KDE and XFCE (although XFCE might get dropped in favor or aarch64 Workstation [3]). We cannot test all of this and honestly claim that we verified basic functionality of all the included apps on all these desktops. I believe we need to adjust the criterion and align it closer with reality. In my eyes, it’s better to have a narrower scope and perform it well than having a large scope and perform it poorly.
*= Proposal =*
Change the criterion to something along these lines:
All applications that can be launched using the standard graphical
mechanism after a default installation of Fedora Workstation on x86_64 architecture must start successfully and withstand a basic functionality test.
For other release-blocking desktops (on any architecture), the requirements only apply to the following types of applications:
web browser (e.g. firefox) [4]
file browser (e.g. nautilus)
package manager (e.g. gnome-software)
image viewer (e.g. eog)
document viewer (e.g. evince)
text editor (e.g. gedit)
archive manager (e.g. file-roller)
terminal emulator (e.g. gnome-terminal) [4]
problem reporter (e.g. abrt)
If there are multiple applications of the same type (e.g. several web browsers), only one of them needs to satisfy the requirements.
As you can see, the original criterion was kept for Fedora’s flagship desktop edition, the one that is most prominent on https://getfedora.org and probably the one that most newcomers download. We would still verify everything on Fedora Workstation on x86_64. But any other desktop (including Workstation on aarch64) would get just reduced criteria, because we simply can’t ensure the same quality bar for the smaller desktop editions/spins. There are some high-profile types of applications that I considered including in the list above, but didn’t in the end:
* word editor (e.g. libreoffice-writer)
* spreadsheet editor (e.g. libreoffice-calc)
* video player (e.g. totem)
* help viewer (e.g. yelp)
I’d like to hear your thoughts on whether they should be included or not. Of course from an end-user point of view, it would be beneficial. But the question is whether we as QA can promise their testing. And also whether we want to block the release e.g. if Gnumeric is broken on armhfp XFCE or if totem doesn’t work on aarch64. Yes, it’s unpleasant, but people using alternative desktops and architectures are usually far from beginners. It’s usually not difficult to install a different application. Also note that the apps included in x86_64 Workstation will be thoroughly tested so they should be very likely to work well even on other architectures (minus some arch-specific issues).
I’d like to target Fedora 32 with this proposal, which means we should decide on this proposal fast. (I wanted to propose it much sooner, but I was waiting on clarifications about new release-blocking images and also on some fesco tickets [3], some of which is still not clarified; so I’m sorry about a late proposal).
Please comment, thanks.
[0] https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Default_appl...
[1] https://fedoraproject.org/wiki/Releases/32/ReleaseBlocking
[2] https://pagure.io/fedora-iot/issue/23
[3] https://pagure.io/fesco/issue/2339
[4] These two are required to work even in Basic release criteria [5], but I decided to include them anyway to avoid confusion (“why is the web browser missing?”), and to make it clear that they need to satisfy the basic functionality (whereas for Basic release criteria just the barebone functionality might be deemed enough)
[5] https://fedoraproject.org/wiki/Basic_Release_Criteria#Required_applications
On Tue, Feb 18, 2020 at 12:45 PM Kamil Paral kparal@redhat.com wrote:
Yesterday I sent out a QA-related proposal that directly affects the release criteria for KDE x86_64 and Workstation x86_64 and (probably) aarch64 images. The proposal is attached below and also available in the test list (including some follow-ups which are recommended to read):
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/t...
Can you please review that proposal and post your thoughts, if you have any? (Responding to test list is preferred, so that we can discuss it in a single place, but responding here is fine as well). Thank you.
Anyone? Yes/No/We don't care?
On Fri, Feb 21, 2020 at 5:48 AM Kamil Paral kparal@redhat.com wrote:
On Tue, Feb 18, 2020 at 12:45 PM Kamil Paral kparal@redhat.com wrote:
Yesterday I sent out a QA-related proposal that directly affects the release criteria for KDE x86_64 and Workstation x86_64 and (probably) aarch64 images. The proposal is attached below and also available in the test list (including some follow-ups which are recommended to read): https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/t...
Can you please review that proposal and post your thoughts, if you have any? (Responding to test list is preferred, so that we can discuss it in a single place, but responding here is fine as well). Thank you.
Anyone? Yes/No/We don't care?
Hi Kamil, You know more about what QA has the resourced to test than I do, so part of me feels I shouldn't be speaking up. But since you asked, twice, I will. First off, thank you for testing all these, you help make Fedora a great distribution to use.
I think cutting the criteria down for KDE and XFCE to just those certain applications is good. That would allow KDE to add or remove applications to our default install, without being overly burdensome on the QA team. It would also set a criteria, that we have to have certain things in our release.
As for those extra applications that you are debating about: I figure the word processor (libre-office) would have already been checked when you check the Workstation desktop, so I don't feel it has to be checked again. The help application, I think should be checked. I would hope that would be something that is easy to check, and it's something most people expect to be there. I know my cat likes hitting my F1 key and pulling up help. video player, I'm not sure either way. Although it's nice ... for me, the first thing I do is install VLC. (Not just on my linux machines either). I sorta expect that whatever video player comes with the Operating System, doesn't have all the codecs I want and that it will be a pain to install them. (Again, this isn't just a linux thing)
Please note, that I am not the KDE spin maintainer. Just an end user and packager. If others feel differently, their opinion might have more sway than mine.
Troy
On Fri, 2020-02-21 at 07:25 -0800, Troy Dawson wrote:
As for those extra applications that you are debating about:
I figure the word processor (libre-office) would have already been
checked when you check the Workstation desktop, so I don't feel it has
to be checked again.
It is possible for us to hit desktop-specific issues in applications, it's happened before. So we'd probably still want to do that.
Adam Williamson wrote:
On Fri, 2020-02-21 at 07:25 -0800, Troy Dawson wrote:
As for those extra applications that you are debating about:
I figure the word processor (libre-office) would have already been checked when you check the Workstation desktop, so I don't feel it has to be checked again.
The word processor on the KDE Spin is actually Calligra Words, not LibreOffice Writer, so the testing done on the GNOME Workstation on LibreOffice Writer is of no use to the KDE Spin as shipped.
It is possible for us to hit desktop-specific issues in applications, it's happened before. So we'd probably still want to do that.
There's that too, indeed. But in the word processor case, it is actually a completely different application that definitely needs separate testing.
Kevin Kofler
Kamil Paral wrote:
Anyone? Yes/No/We don't care?
I am opposed to this, as it is means Fedora is more likely to ship with broken KDE applications. Even the core ones if the KDE Spin keeps also shipping, e.g., Firefox (something I think should just stop, but that is not my decision to make), since your proposal only requires ANY browser on the Spin to work. I do not think it is acceptable to ship a Fedora KDE release with a broken Falkon.
I also think that if we really want to limit the kinds of applications that we block on, the same list should also apply to the GNOME-based Workstation. I do not see why that should get preferential treatment over other release- blocking Spins. Release-blocking is release-blocking.
Kevin Kofler
Kamil Paral wrote:
I am opposed to this, as it is means Fedora is more likely to ship with broken KDE applications. Even the core ones if the KDE Spin keeps also shipping, e.g., Firefox (something I think should just stop, but that is not my decision to make), since your proposal only requires ANY browser on the Spin to work. I do not think it is acceptable to ship a Fedora KDE release with a broken Falkon.
Perhaps that could be an impetus to ship just one browser. The current situation is not just a poor situation for QA, it's also a poor situation for users. What I really wanted to avoid is to specify concrete application names on concrete desktops that need to be checked. That is a maintenance nightmare. That's why I suggested generic solution. The app duplication is not that common, it occurs mainly just on KDE.
I also think that if we really want to limit the kinds of applications that we block on, the same list should also apply to the GNOME-based Workstation. I do not see why that should get preferential treatment over other release- blocking Spins. Release-blocking is release-blocking.
So you're OK with that change as long as no one else gets a better deal? Hmm. I've specified the reason. This is not about release-blocking status. Both Server and Workstation are release blocking, but e.g. broken ssh or remote logging would only block on Server and not on Workstation. We don't have universal rules across all release artifacts. This is about Workstation being a flagship product, an Edition, an artifact that gets offered on the very home page of getfedora.org. As such, we can have extra requirements for it. KDE is a Spin, a lower visibility artifact. You might be salty about that, but I'm just stating the facts, this is not an attack on the quality or usefulness of KDE itself.
Don't get me wrong, I'd love to have all images that we produce to have a top-notch quality. But our team has a fixed size, the number of important products keeps growing, we don't see much external involvement in release validation efforts, and we need to resolve it somehow.
Out of curiosity, would you be willing to contribute in release validation testing and cover the apps which are not part of the proposal, in order to keep the "all apps must work" criterion for the KDE spin? I'm not sure it is the right solution, but I'm interested in hearing your thoughts.
On Mon, Feb 24, 2020 at 11:11 AM Kamil Páral kparal@fedoraproject.org wrote:
I am opposed to this, as it is means Fedora is more likely to ship with broken KDE applications.
Hello, I have been doing the KDE "all applications must work" test for more than two years already and I can tell you, that some of the KDE pre-installed applications are of low quality, if not broken already. Last time I checked, there were over 60 applications listed in Menu where there are like 30 in Gnome. Some applications (browsers) are tripled, some are doubled. Others are connected with a certain use case, e.g. converting one data format into another, therefore useless for people without that use case.
If "all applications" must work, than I need to start every application and test the "basic functionality". Nowhere is written exactly, what basic functionality is, so I must test what I think needs to be tested "in bona fide." Many KDE applications generally work, but they have flaws in some parts that can or do not need to be basic functionality, depending on opinions.
In case of a reported bug, the readiness of KDE developers to fix it usually is lower when compared to Gnome, so sometimes there are bugs that will never be fixed, because they are not as severe as to block the release completely, so they get ship over and over again.
The proposal could actually improve the situation, because important apps would be tested thoroughly and not just for the basic functionality. We do not want to order which applications that should be, I think the SIGs should create a list of such applications. What the proposal says is just a generic example of applications that "make sense" on a desktop system.
We lack help on testing desktops and we probably would not want to solve this, if there were plenty of community members doing the applications tests and reporting to the matrices. In the past, there were moments when we had to retest a new compose like 12 hours prior to the releas readiness meeting and we had to assign one person to work on KDE applications only for half the time.
Have a nice day, Kevin.
Lukas
Even the core ones if the KDE Spin keeps also
shipping, e.g., Firefox (something I think should just stop, but that is
not
my decision to make), since your proposal only requires ANY browser on
the
Spin to work. I do not think it is acceptable to ship a Fedora KDE
release
with a broken Falkon.
The responsiveness to KDE related bugs is much lower (when compared to Gnome), so even if a bug is reported, it will not get fixed so easily, or possibly ever.
Out of curiosity, would you be willing to contribute in release validation testing and cover the apps which are not part of the proposal, in order to keep the "all apps must work" criterion for the KDE spin? I'm not sure it is the right solution, but I'm interested in hearing your thoughts. _______________________________________________ kde mailing list -- kde@lists.fedoraproject.org To unsubscribe send an email to kde-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kde@lists.fedoraproject.org
On Mon, Feb 24, 2020 at 12:08 PM Lukas Ruzicka lruzicka@redhat.com wrote:
Hello, I have been doing the KDE "all applications must work" test for more than two years already and I can tell you, that some of the KDE pre-installed applications are of low quality, if not broken already. Last time I checked, there were over 60 applications listed in Menu where there are like 30 in Gnome. Some applications (browsers) are tripled, some are doubled. Others are connected with a certain use case, e.g. converting one data format into another, therefore useless for people without that use case.
If "all applications" must work, than I need to start every application and test the "basic functionality". Nowhere is written exactly, what basic functionality is, so I must test what I think needs to be tested "in bona fide." Many KDE applications generally work, but they have flaws in some parts that can or do not need to be basic functionality, depending on opinions.
In case of a reported bug, the readiness of KDE developers to fix it usually is lower when compared to Gnome, so sometimes there are bugs that will never be fixed, because they are not as severe as to block the release completely, so they get ship over and over again.
The proposal could actually improve the situation, because important apps would be tested thoroughly and not just for the basic functionality.
Lukas described it all nicely, but in this particular place I have to clarify. The proposed criterion still requires just "basic functionality" of covered apps. If we can use some of the saved time to test those covered applications more thoroughly is a different question, and it's possible, but I don't want to promise anything. Another possible improvement is that we might be able to automate testing of some of those applications.
I want to make one additional point. I could have proposed that the criterion applied to all desktops. From QA point, it's less work for us, that should make us happy, right? But I didn't - because I'm trying to find some balance between our responsibilities and the end-user benefits. And it's a difficult balance to find. If we can't find it, that easy solution is always there. If you have opinions on where to draw the line and how to improve the situation in a different way, please tell us.
-
Adam Williamson -
Kamil Paral -
Kamil Páral -
Kevin Kofler -
Lukas Ruzicka -
Troy Dawson