4:06 a.m.
Hello KDE and XFCE teams,
I am writing to you on behalf of the Fedora QA team, because you are the
stakeholders of release-blocking desktop environments in Fedora.
We believe that there is need to revisit and review some of the approaches
we
have in desktop testing. I hope that I can clarify in the text.
*How do we test desktop applications?*
Currently, there are three desktop environments, which block the Fedora
releases: GNOME, KDE, and XFCE (on ARM). Among others, there is a test case
called “Desktop Menus” [
https://fedoraproject.org/wiki/QA:Testcase_desktop_menus ] which is used
for all three desktop environments.
*What is the purpose of the Desktop Menus test case?*
The purpose of the “Desktop Menus” test case is to test that applications
listed
in the menu do generally work. That means that each application in the
default
system menu must launch without crashing and withstand basic usage. [2]
*What problems can arise when trying to follow the test case?*
This approach leaves a lot of space for variability, because the terms are
not
clearly given, especially when talking about “basic functionality” of the
desktop applications. In the past, there have been some disputes during
Blocker
Review meetings, or during Go/No go meetings whether basic functionality is
broken or not. It's usually up to group judgement to decide what is a basic
functionality and what it beyond that, which means as testers we need to
err on
the side of doing more testing rather than less.
Testing the “Desktop Menus” test case is one of the most time consuming test
case we currently have. In GNOME, there are currently about 40 applications
in
the default menu and all need to be started and their basic functionality
tested. In KDE, the number of applications is even higher, and some
application
types (like web browsers) are even present as several different
applications.
The scope of this test case also covers testing the subcomponents in the
Control
Center (or a similar equivalent) and, usually, such subcomponents are
numerous.
Testing XFCE on ARM is also rather time consuming, because the ARM devices
we
can use for testing are very slow and there are some application that make
little sense on an ARM motherboard (e.g. a CD writing software). Thus,
doing a
thorough Desktop Menus test case requires a rather longish time frame and we
lose time we could devote to different problems.
Besides that, the test case is also very boring, so it usually gets tested
by
Fedora QA members only, with few inputs from the community. So, before the
Beta
and Final release we are able to do such tests a couple of times, but the
overall coverage is not big.
*What could we do about it?*
At Fedora QA, we have been trying to come up with some ideas to make the
situation a little better than it is now. The goal is that the important
areas
get enough attention and testing they deserve, and we avoid shipping broken
and
under-tested software. See the ideas below and please tell us if you could
adopt
all/some of them, or if you have different suggestions to improve the
current
state, we'd love to hear them too.
- The list of pre-installed applications could be pruned of those which
don't make much sense for that particular purpose (e.g. CD writing software
on ARM, or perhaps on any system nowadays).
- Pre-installed applications could be deduplicated where possible (e.g.
not having several web browsers pre-installed by default).
- We could define a list of "important" applications that would be
tested thoroughly and the basic functionality criterion would apply just to
them. That would tell QA where to focus. (This would obviously contain
applications like system settings, file browser, text editor, browser,
terminal emulator, etc. But e.g. a screenshot utility would probably not
fall into the list. Also, in case of duplicated apps, only one of those
could be marked as "important", and the others would* not). The rest of the
applications would be just best effort in terms of QA, and wouldn't block
the release.
- Last but not least, you could help us more with release validation
before relevant milestones (Beta, Final) :-) If there's anything that
should be improved on our part (communication, instructions, etc), we'd
love to hear that.
Please, let me know what you think about the ideas proposed above. Do you
have
any other ideas that could help us improve the quality (with the same
amount of
testing force)?
Best regards,
--
Lukáš Růžička
FEDORA QE, RHCE
Red Hat
<https://www.redhat.com>
Purkyňova 115
612 45 Brno - Královo Pole
lruzicka(a)redhat.com
TRIED AND PERSONALLY TESTED, ERGO TRUSTED. <https://redhat.com/trusted>