On Tue, Jan 8, 2019 at 11:08 AM Lukas Ruzicka <lruzicka@redhat.com> wrote:
Hello KDE and XFCE teams,

I am writing to you on behalf of the Fedora QA team, because you are the
stakeholders of release-blocking desktop environments in Fedora.

We believe that there is need to revisit and review some of the approaches we
have in desktop testing. I hope that I can clarify in the text.

How do we test desktop applications?

Currently, there are three desktop environments, which block the Fedora
releases: GNOME, KDE, and XFCE (on ARM). Among others, there is a test case
called “Desktop Menus” [ https://fedoraproject.org/wiki/QA:Testcase_desktop_menus ] which is used for all three desktop environments.

Why is Xfce tested only on ARM devices? I mean if that's one of the main reasons it's slow and tedious, why not change the architecture?
Additionally, it would be probably worthwhile to create two different sets of programs, one group has to start and thus needs to be tested and the other is a nice
to have, but doesn't need to start. At least the criteria could be a little bit less strict.

Johannes

What is the purpose of the Desktop Menus test case?

The purpose of the “Desktop Menus” test case is to test that applications listed
in the menu do generally work. That means that each application in the default
system menu must launch without crashing and withstand basic usage. [2]

What problems can arise when trying to follow the test case?

This approach leaves a lot of space for variability, because the terms are not
clearly given, especially when talking about “basic functionality” of the
desktop applications. In the past, there have been some disputes during Blocker
Review meetings, or during Go/No go meetings whether basic functionality is
broken or not. It's usually up to group judgement to decide what is a basic
functionality and what it beyond that, which means as testers we need to err on
the side of doing more testing rather than less.

Testing the “Desktop Menus” test case is one of the most time consuming test
case we currently have. In GNOME, there are currently about 40 applications in
the default menu and all need to be started and their basic functionality
tested. In KDE, the number of applications is even higher, and some application
types (like web browsers) are even present as several different applications.
The scope of this test case also covers testing the subcomponents in the Control
Center (or a similar equivalent) and, usually, such subcomponents are numerous.

Testing XFCE on ARM is also rather time consuming, because the ARM devices we
can use for testing are very slow and there are some application that make
little sense on an ARM motherboard (e.g. a CD writing software). Thus, doing a
thorough Desktop Menus test case requires a rather longish time frame and we
lose time we could devote to different problems.

Besides that, the test case is also very boring, so it usually gets tested by
Fedora QA members only, with few inputs from the community. So, before the Beta
and Final release we are able to do such tests a couple of times, but the
overall coverage is not big.

What could we do about it?

At Fedora QA, we have been trying to come up with some ideas to make the
situation a little better than it is now. The goal is that the important areas
get enough attention and testing they deserve, and we avoid shipping broken and
under-tested software. See the ideas below and please tell us if you could adopt
all/some of them, or if you have different suggestions to improve the current
state, we'd love to hear them too.


  • The list of pre-installed applications could be pruned of those which don't make much sense for that particular purpose (e.g. CD writing software on ARM, or perhaps on any system nowadays).
  • Pre-installed applications could be deduplicated where possible (e.g. not having several web browsers pre-installed by default). 
  • We could define a list of "important" applications that would be tested thoroughly and the basic functionality criterion would apply just to them. That would tell QA where to focus. (This would obviously contain applications like system settings, file browser, text editor, browser, terminal emulator, etc. But e.g. a screenshot utility would probably not fall into the list. Also, in case of duplicated apps, only one of those could be marked as "important", and the others would* not). The rest of the applications would be just best effort in terms of QA, and wouldn't block the release. 
  • Last but not least, you could help us more with release validation before relevant milestones (Beta, Final) :-) If there's anything that should be improved on our part (communication, instructions, etc), we'd love to hear that.
Please, let me know what you think about the ideas proposed above. Do you have
any other ideas that could help us improve the quality (with the same amount of
testing force)?


Best regards,

--

Lukáš Růžička

FEDORA QE, RHCE

Red Hat

Purkyňova 115

612 45 Brno - Královo Pole

lruzicka@redhat.com   

_______________________________________________
xfce mailing list -- xfce@lists.fedoraproject.org
To unsubscribe send an email to xfce-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/xfce@lists.fedoraproject.org