Hello, KDE on aarch64 became a blocking deliverable this cycle, at KDE team's request [1]. The QA team sadly doesn't have the capacity to test it regularly, and so it was agreed that it would be the KDE team's responsibility. I talked to Neal Gompa today and he told me that the QA processes are hard to discover and it's one of the reasons why no test results were submitted to our release validation matrices, even for F41 Beta. In this email, I'd like to provide guidance on how to navigate QA matrices and hopefully make the process clearer and easier for you to contribute :-)
I'll first showcase the test matrices on F41 Beta, and later I'll show how to discover the "current" test matrix and figure out what needs testing.
== How to provide test results ==
Specifically for KDE on aarch64, the most relevant sections of our matrices are these: 1. https://fedoraproject.org/wiki/Test_Results:Fedora_41_Beta_1.2_Installation#... 2. https://fedoraproject.org/wiki/Test_Results:Fedora_41_Beta_1.2_Desktop#Relea... 3. https://fedoraproject.org/wiki/Test_Results:Fedora_41_Beta_1.2_Base#Release-...)
You can see the KDE column (or a row) in all of these tables. The Milestone column shows when exactly this feature is supposed to be fully functional (when there are multiple milestones, usually you can see links inside the test case to multiple release criteria related to multiple milestones), the Test Case column leads you to test steps description. (Note: the "coconut" results are provided by our test automation framework).
For a Beta/Final release candidate images (RCs), all the included test cases (not marked as Optional in the Milestone column) are supposed to be tested and their results filled out. For interim nightly images, we're supposed to perform some reasonable coverage in between, e.g. test all Final test cases at least once before a Final RC actually arrives, to identify bugs as early as possible. It's also a good idea to re-test stuff after a big system update (e.g. a new KDE release, etc).
At the beginning of each page, there are links to download composed images, and a section describing how to edit the wiki page and provide test results.
== How to know which test matrix to fill out ==
There are several ways on how to learn which compose to test and where to fill out the results:
1. We announce all "important composes" (all RCs and some nightly composes which we decided to nominate for collaborative testing) in our test-announce mailing list. Search for "candidate" and "nightly compose" in email subjects: https://lists.fedoraproject.org/archives/list/test-announce@lists.fedoraproj...
2. All old (non-current) wiki pages have a link at the very top leading to the latest test matrix page. See e.g. the Beta pages linked above.
3. There's a handy category containing all Fedora 41 test matrices here: https://fedoraproject.org/wiki/Category:Fedora_41_Test_Results At the top, you can see links which always lead to the latest test matrices (the Summary one displays all of them as a single page). Below that, you can see subcategories for Nightly/Beta/Final (when available) test matrices, which show all pages created during this cycle.
== Which test cases to focus on? ==
For RCs, it's simple, everything should be tested and filled out (as described above). As for nominated composes, we can't test them all every time, and so there's an obvious question "What should I focus on? When was this test case tested last, so that I can retest the oldest results?". We have an ugly but functional tool for that called testcase_stats:
https://openqa.fedoraproject.org/testcase_stats/
Here's an example, the F41 Desktop testing stats:
https://openqa.fedoraproject.org/testcase_stats/41/Desktop.html
See the explanatory key at the top.
However, it currently only shows stats per each testcase (row in a matrix), but not also combined per each environment (column, e.g. Workstation vs KDE). For this specific use case (looking just at KDE results), it's not very handy. You can always click on Details for a particular test case and see if KDE was tested or not and when, but it gets time-consuming quickly. Here's an example of "Release-blocking desktops: aarch64" -> "QA:Testcase_desktop_panel_basic" detailed stats (scroll down):
https://openqa.fedoraproject.org/testcase_stats/41/Desktop/QA_Testcase_deskt...
We'll see if we can improve the UI to make it more useful to teams with some narrow and not global test focus.
== Ask us ==
If there's anything unclear, we'll be happy to explain it better, either here in this kde mailing list, or our test list [2], or on our #quality:fedoraproject.org Matrix channel, please join it!
Also, if you have time to not just test KDE on aarch64, but also help with KDE on x86_64 testing (see the matrices, they are clearly labeled with x86_64), that would be very appreciated!
Cheers, Kamil Fedora QA
[1] https://pagure.io/fesco/issue/3232 [2] https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/