On Tue, 2024-09-24 at 16:36 +0200, Peter Boy wrote:
A good starting page would be the server page that is now sent in the announcement emails: https://fedoraproject.org/wiki/Test_Results:Fedora_41_Branched_20240924.n.0_...
We should remove all items that have nothing to do with Server, starting with the download list.
I disagree with this. The download list is there for a reason: when we didn't have it, we sent people to the pages, and they said "but where do I download the thing to test in the first place?" So we added the download lists.
What we *could* do is set the table up to be filtered on edition specific pages - only show Server images on the Server page, only show desktop images on the Desktop page etc. I can try and find some time to look into that (though I have about 40 things I'm trying to find time to look into already ATM :|). If anyone else wants to look, the code is https://pagure.io/fedora-qa/python-wikitcms/blob/main/f/src/wikitcms/page.py... . The fancy way to do it would be to add some kinda magic to the table to let different pages have different 'views' on it or something (I don't know off the top of my head if this is possible). The dumb-but- probably-easy way to do it would just be to generate several different tables, either as separate pages or all in one page in different sections set up to be transcluded by different test pages.
The test matrix and coverage page / lists should be split into automated tests and manual/human tests.
I don't really agree with this either, except possibly for cases where a test is human-run *by policy*. Otherwise these are not inherent characteristics of the test, but implementation details. Automation has happened gradually; at some point, *all* the tests in the matrix were human-run. We only automated some of the tests on the Server page fairly recently.
We do already distinguish test results from automated systems - they're the ones with the robot icon next to them. To me this is the right way to do it: automation is a property of the *result*, not of the *test*. Results from automation should always be posted within a few hours after the validation event is created, and you can always refer to past result pages to see which tests get results filed by automation.
Note we *do* already explicitly require a smoke test install of Server (along with all of the other release-blocking images) to real hardware using a real USB stick. This is in the Installation page, not the Server page - https://fedoraproject.org/wiki/Test_Results:Fedora_41_Branched_20241012.n.0_...) .
We *could* break out some rows of that table onto the various edition- specific pages, but it gets kinda awkward since Everything is in the list and there's no page for Everything. We'd need an instance of the table on multiple subpages *and* to keep it on Installation for Everything, which I don't really love. I do get the point that, if you specifically want to "help with testing Server", it's a bit awkward that you have to find or know that there are Server-relevant tests on the Installation and Base pages too. I'll try and think of ways to improve that.
Just for some context, the reason I want to emphasize use of the current setup is it's already the result of a process that started with more or less what you started with (ad-hoc, human created wiki pages). The very first one (I think) was https://fedoraproject.org/wiki/Releases/5/FC5FinalTreeTesting . The current wikitcms setup is a direct (though distant) descendant of that page. :D
It seems easy at first to just throw up a simple wiki page with some tables in it. Then you get more tests. Then people start asking questions like "where do I download this thing? How do I enter results exactly?". Then you want to get some data on the results over time. Then you start getting really freaking tired of manually creating these damn wiki pages every three days.
For instance, in your current page, you don't cover arches. You don't have a page per build, which makes comparing results from different builds and knowing which builds are "covered" difficult. You have the comments in-line in the tables, which will make the tables very hard to read if anyone puts a long comment in (this is why, on the wikitcms pages, the comments are footnotes). etc etc etc...this is all stuff we went through already, years ago. :D
We went through that whole cycle, and the current wikitcms setup is the end point of it. I'd really hate someone to have to go through that whole cycle again from scratch.