On Tue, 2022-05-03 at 19:37 -0500, Michael Catanzaro wrote:
On Tue, May 3 2022 at 05:14:53 PM -0700, Adam Williamson adamwill@fedoraproject.org wrote:
Stable releases of core components of a major desktop should never contain bugs like "deleting contacts sometimes doesn't work" or "you can't add photos to an album in the Photos application because the dialog where you're supposed to do it is completely broken and the list entries multiply like rabbits who've been dosed up on viagra". Distribution validation testing is not *for* finding bugs like this.
I kinda agree, but reality is GNOME has no QA, and Fedora has good QA, so Fedora is gonna find the bugs.
(IMO GNOME quality has actually improved considerably over the past decade. But surely not enough so!)
GNOME, yes. The thing this makes me wonder is: does it *really* make sense for GNOME to be shipping all these desktop apps, if it doesn't have the resources to properly maintain or test them?
Do GNOME users actually use Contacts and Calendar and Photos? I mean, I'm kind of a GNOME ultra, and I don't. I still use Evolution because it still works and does everything I want and I'm used to it. I still use Shotwell because all my metadata is in it and there doesn't seem to be any way to export it to Photos, so why would I switch?
We (Fedora) are more likely to happen across bugs in the core of GNOME than in apps nobody really uses, if that's the kind of QA upstream GNOME is relying on now.
GNOME does have OpenQA running now, but there are almost no tests. Maintaining OpenQA tests requires effort that nobody has been interested in. I don't want this to sound like an invitation "adamw come prepare and maintain loads of OpenQA tests for us! do even more work! don't you want a second unpaid job?" but if anybody happens to think this would be an interesting way to contribute, pushing existing tests further upstream would not be a bad idea.
We could look at what it would take to upstream the app tests we have already for Fedora. What I'm afraid of is that we'd wind up finding some subtle difference in theme or font rendering meant we'd have to duplicate all the needles, and re-do them all twice every time freetype or cantarell sneezes...
On Tue, May 03, 2022 at 09:20:35PM -0700, Adam Williamson wrote:
We could look at what it would take to upstream the app tests we have already for Fedora. What I'm afraid of is that we'd wind up finding some subtle difference in theme or font rendering meant we'd have to duplicate all the needles, and re-do them all twice every time freetype or cantarell sneezes...
So, um, maybe this is another place we could talk with GNOME upstream. Could we have those upstream tests run on Rawhide?
Also: where does GNOME _expect_ the human-based QA to happen? Especially for the applications, it's going to be hard to automate thoroughly. Maybe we can figure out something better there too.
On Wed, 2022-05-04 at 19:04 -0400, Matthew Miller wrote:
On Tue, May 03, 2022 at 09:20:35PM -0700, Adam Williamson wrote:
We could look at what it would take to upstream the app tests we have already for Fedora. What I'm afraid of is that we'd wind up finding some subtle difference in theme or font rendering meant we'd have to duplicate all the needles, and re-do them all twice every time freetype or cantarell sneezes...
So, um, maybe this is another place we could talk with GNOME upstream. Could we have those upstream tests run on Rawhide?
Also: where does GNOME _expect_ the human-based QA to happen? Especially for the applications, it's going to be hard to automate thoroughly. Maybe we can figure out something better there too.
So the answer to both those questions is, AIUI, kinda the same: GNOME wants the testing run on GNOME OS.
which is (supposed to be, I don't know how closely it meets the goal ATM) a nightly live GNOME image with the latest versions of everything in it.
Testing that is much more useful to upstream than testing Rawhide, because we certainly aren't updating all the bits of GNOME in Rawhide nightly. But if it's not built on Fedora (I don't think it is), it may well have differences in theme and font configuration that might make Fedora's openQA needles not match, which is the problem I'm concerned about with sharing the tests.
On Wed, May 04, 2022 at 05:47:33PM -0700, Adam Williamson wrote:
So, um, maybe this is another place we could talk with GNOME upstream. Could we have those upstream tests run on Rawhide?
Also: where does GNOME _expect_ the human-based QA to happen? Especially for the applications, it's going to be hard to automate thoroughly. Maybe we can figure out something better there too.
So the answer to both those questions is, AIUI, kinda the same: GNOME wants the testing run on GNOME OS.
which is (supposed to be, I don't know how closely it meets the goal ATM) a nightly live GNOME image with the latest versions of everything in it.
Testing that is much more useful to upstream than testing Rawhide, because we certainly aren't updating all the bits of GNOME in Rawhide nightly. But if it's not built on Fedora (I don't think it is), it may well have differences in theme and font configuration that might make Fedora's openQA needles not match, which is the problem I'm concerned about with sharing the tests.
Could we (GNOME and Fedora in collaboration) set up something where they _are_ updating everything on top of Rawhide nightly? If not instead of GNOME OS, at least alongside it?
That seems like it would:
1) Give a better practical test environment for GNOME folks
2) Make the needles match
3) Maybe make it so developers can fix their own needle-breaking problems?
4) Help everything get tested earlier
On Thu, May 5, 2022 at 1:23 PM Matthew Miller mattdm@fedoraproject.org wrote:
Wading into this discussion pretty late... there's far too much "GNOME does this" or "GNOME wants that" in this discussion.
That is really not how it works, neither for GNOME nor for Fedora - its down to small teams and individual maintainers to make changes and decisions. For improving the QA situation between upstream GNOME and downstream Fedora, a good team to talk to is the GNOME release team of which both Michael and I happen to be members.
Testing that is much more useful to upstream than testing Rawhide, because we certainly aren't updating all the bits of GNOME in Rawhide nightly. But if it's not built on Fedora (I don't think it is), it may well have differences in theme and font configuration that might make Fedora's openQA needles not match, which is the problem I'm concerned about with sharing the tests.
Could we (GNOME and Fedora in collaboration) set up something where they _are_ updating everything on top of Rawhide nightly? If not instead of GNOME OS, at least alongside it?
This would be a useful discussion to have, Having it in a bof at guadec sounds great.
On Fri, May 6, 2022 at 7:29 AM Matthias Clasen mclasen@redhat.com wrote:
On Thu, May 5, 2022 at 1:23 PM Matthew Miller mattdm@fedoraproject.org wrote:
Wading into this discussion pretty late... there's far too much "GNOME does this" or "GNOME wants that" in this discussion.
That is really not how it works, neither for GNOME nor for Fedora - its down to small teams and individual maintainers to make changes and decisions. For improving the QA situation between upstream GNOME and downstream Fedora, a good team to talk to is the GNOME release team of which both Michael and I happen to be members.
Testing that is much more useful to upstream than testing Rawhide, because we certainly aren't updating all the bits of GNOME in Rawhide nightly. But if it's not built on Fedora (I don't think it is), it may well have differences in theme and font configuration that might make Fedora's openQA needles not match, which is the problem I'm concerned about with sharing the tests.
Could we (GNOME and Fedora in collaboration) set up something where they _are_ updating everything on top of Rawhide nightly? If not instead of GNOME OS, at least alongside it?
This would be a useful discussion to have, Having it in a bof at guadec sounds great.
One of the things we're working on in the KDE SIG is eventually auto-building all our KDE software in a COPR on a regular schedule, with the goal of eventually producing something that we could wire up to run through OpenQA tests using that COPR repo for precisely this purpose. I think it would absolutely make sense if the folks maintaining GNOME did the same thing, since they're both release blocking desktops.
-
Adam Williamson -
Matthew Miller -
Matthias Clasen -
Neal Gompa