Am 14.10.2024 um 18:31 schrieb Adam Williamson via server server@lists.fedoraproject.org:
On Tue, 2024-09-24 at 16:36 +0200, Peter Boy wrote:
A good starting page would be the server page that is now sent in the announcement emails: https://fedoraproject.org/wiki/Test_Results:Fedora_41_Branched_20240924.n.0_...
We should remove all items that have nothing to do with Server, starting with the download list.
I disagree with this. The download list is there for a reason: when we didn't have it, we sent people to the pages, and they said "but where do I download the thing to test in the first place?" So we added the download lists.
What we *could* do is set the table up to be filtered on edition specific pages - only show Server images on the Server page,
Sorry, that filtering is what I meant. So we have no disagreement.
only show desktop images on the Desktop page etc. I can try and find some time to look into that (though I have about 40 things I'm trying to find time to look into already ATM :|). If anyone else wants to look, the code is https://pagure.io/fedora-qa/python-wikitcms/blob/main/f/src/wikitcms/page.py... . The fancy way to do it would be to add some kinda magic to the table to let different pages have different 'views' on it or something (I don't know off the top of my head if this is possible). The dumb-but- probably-easy way to do it would just be to generate several different tables, either as separate pages or all in one page in different sections set up to be transcluded by different test pages.
Yes, for me I would we content if have a solution we want to build. Maybe it takes some time to get the resources to make it real. In the meantime, we can live with a makeshift solution.
The test matrix and coverage page / lists should be split into automated tests and manual/human tests.
I don't really agree with this either, except possibly for cases where a test is human-run *by policy*. Otherwise these are not inherent characteristics of the test, but implementation details. Automation has happened gradually; at some point, *all* the tests in the matrix were human-run. We only automated some of the tests on the Server page fairly recently.
OK, I See from a QA group view it makes not much sense to differentiate the tables.
But from a WG view we would need a clear and consistent table of what we need or should do. So probably your other idea, to provide a table of links to the tests in question might be good solution?
We do already distinguish test results from automated systems - they're the ones with the robot icon next to them. To me this is the right way to do it: automation is a property of the *result*, not of the *test*. Results from automation should always be posted within a few hours after the validation event is created, and you can always refer to past result pages to see which tests get results filed by automation.
I totally agree with that.
Note we *do* already explicitly require a smoke test install of Server (along with all of the other release-blocking images) to real hardware using a real USB stick. This is in the Installation page, not the Server page - https://fedoraproject.org/wiki/Test_Results:Fedora_41_Branched_20241012.n.0_...) .
We *could* break out some rows of that table onto the various edition- specific pages, but it gets kinda awkward since Everything is in the list and there's no page for Everything. We'd need an instance of the table on multiple subpages *and* to keep it on Installation for Everything, which I don't really love. I do get the point that, if you specifically want to "help with testing Server", it's a bit awkward that you have to find or know that there are Server-relevant tests on the Installation and Base pages too. I'll try and think of ways to improve that.
That’s the issue, indeed! And I would love it you can make a proposal how to do it in a good way.
Just for some context, the reason I want to emphasize use of the current setup is it's already the result of a process that started with more or less what you started with (ad-hoc, human created wiki pages). The very first one (I think) was https://fedoraproject.org/wiki/Releases/5/FC5FinalTreeTesting . The current wikitcms setup is a direct (though distant) descendant of that page. :D
I wholeheartedly support that. My original idea was to create a “test week” for Server, too. This would have everything that needs to be done in one place and very clearly laid out, with links to the corresponding tests in the overall view.
But you said that it couldn't be done with the available resources.
It seems easy at first to just throw up a simple wiki page with some tables in it. Then you get more tests. Then people start asking questions like "where do I download this thing? How do I enter results exactly?". Then you want to get some data on the results over time. Then you start getting really freaking tired of manually creating these damn wiki pages every three days.
For instance, in your current page, you don't cover arches. You don't have a page per build, which makes comparing results from different builds and knowing which builds are "covered" difficult. You have the comments in-line in the tables, which will make the tables very hard to read if anyone puts a long comment in (this is why, on the wikitcms pages, the comments are footnotes). etc etc etc...this is all stuff we went through already, years ago. :D
We went through that whole cycle, and the current wikitcms setup is the end point of it. I'd really hate someone to have to go through that whole cycle again from scratch.
Again, I'm not committed to this table and I'm not happy with it. I see it as a stopgap because we haven't been able to create anything decent so far. Since the revival of the Server Working Group, we have stumbled from one temporary solution to the next. I want to move away from the unsystematic way we have been doing things so far. I'm not the expert on distribution QA. I'm just the fool trying to organize the necessary testing to get done, to make the best of the unsatisfactory situation, and to initiate the discussion about who of us is doing what.
As a first step, we should agree on what tests we want and need to do systematically and reasonably. You are the member of our working group who knows the most about this. So it would be best if you put together a to-do list on this.
If I understand our discussion so far correctly, that might be something like:
1) Smoke tests of the installation media (all those „USB“ tests)
2) Possibly test of server-specific procedures, e.g. installation of virtualization with internal protected network and internal name resolution (but maybe this can also be automated) or our special web server configuration
3) Review of all changes to a release to see if a change might particularly challenge Server (such as the aforementioned LVM change if it had been announced).
But you are the expert, not me.
And in a second step we should find a way to make it easy for every WG member to participate in the testing, probably by modifying the Server test announce page a bit.
And in a third step, we have to gradually supplement the automatic tests as far as possible by the central services that we have compiled in our technical specification.
Can you participate at the meeting later today? Would be very helpful. I would really like to bring this point to a conclusion, which has been occupying us for a long time.
-- Peter Boy https://fedoraproject.org/wiki/User:Pboy PBoy@fedoraproject.org
Timezone: CET (UTC+1) / CEST (UTC+2)
Fedora Server Edition Working Group member Fedora Docs team contributor and board member Java developer and enthusiast