Sending email from Pavlin because he is not a member of the group and
message is rejected.
--
QUESTIONS
How do we test desktop applications?
On what architectures do we test?
What problems can arise when trying to follow the test case?
I will provide you with brief answer.
WHAT DO WE TEST?
We do automated testing of GNOME environment ONLY.
GUI applications from GNOME fall into our domain of tests.
Our team do not develop automated tests for other GUI environments.
WHERE DO WE TEST?
We test GNOME GUI on several CPU platforms:
1. x86_64
2. PPC64
3. PPC64LE
4. Aarch64 (ARM)
5. s390x
The above statement is valid for RHEL 7.
In the case of RHEL 8, those platforms are reduced to:
1. x86_64
2. PPC64LE
In all cases we test on machines available in Beaker.
For all tests without specific hardware requirements we use VM.
For testing Wi-Fi and displays, we use physical hosts.
HOW DO WE TEST?
We create automated tests which have several major parts:
1) define the logic of a test
2) prepare conditions for testing
3) define functions and algorithms
4) execute GUI testing
5) check for results
(1)
To describe the logic of a test we use English sentences in plain text
files.
This language is called /*Behave*/ (Gherkin language).
Each sentence is connected to a programming function.
(2)
To prepare initial conditions for tests we use:
1. YAML file - contains a list of needed packages
2. a bash script - to set initial conditions for all tests
3. a python module - to set specific conditions for some tests
This preparation includes but is not limited to:
* install needed RPM packages
* install needed Python modules
* set up system parameters
* download security certificates
* add polkit privilege to allow passwordless setup in some components
* grant sudo privilege to users
* change passwords
* restart system services
* set up printers
* set up network interfaces and connections
* set up VPN connections
* set up 802.1x security
(3)
Functions that perform GUI testing are written in Python.
Each sentence from the logical language Behave has its corresponding
function in Python.
This correspondence is known as "/*step*/". It's the "link area"
between
Python and Behave.
Every parameter from an English sentence (in Behave) becomes a function
parameter in Python.
Thus, a sentence from the human language is transformed into programming
code.
To test RHEL 7 we use Python 2.7.
To test RHEL 8 we use Python 3.6.
(4)
To make automated tests manage GUI we use:
1. Accessibility Technology (A11Y)
2. dogtail
3. Python
Combining those we are able to perform various tasks in the GUI.
An automated test clicks/double clicks/drags/types text in GUI components.
Most frequently used components:
* frame windows
* dialog windows
* text labels
* edit fields
* buttons
* combo boxes
* check boxes
* tables
Since the GUI hierarchy is based on parent-child relationship, we use
that relationship for testing.
/*dogtail*/ "sees" a GUI element on the screen via /*A11Y*/ technology.
dogtail performs various activities on different GUI elements in the
same ways as a human user would do.
(5)
After the testing of GUI components is complete
and modifying of system settings is done
we check for results.
The expects results are found in the modified system state.
We check a specific area of RHEL by using shell command or a function in
Python.
*Examples*
1. An automated test starts an application then it checks if it's in
the output of "ps ax".
2. An automated test makes a change in GNOME environment then it
searches for that change in GSettings.
3. An automated test changes the user's password then it tries to log
in with those credentials.
What problems can arise when trying to follow the test case?
Most common issues in our automated tests:
1. Initial conditions for tests are not prepared.
2. The tested GUI application has new interface, different placement of
GUI elements. The current automated test does not work.
3. The tested GUI application has new behaviour. The current automated
test does not work.
4. Unexpected side effects come out which block the execution. For
instance delays, clicking before a GUI form appears, unable to click
on a GUI element, unable to make changes, unable to scroll down the
GUI, etc.
5. A bug has appeared in the tested application. In that case testing
can continue after bug resolution.
6. A bug has appeared in some of the dependent packages. Similarly, the
automated test works after the bug is fixed.
7. The sequence of steps is incorrect. It must be changed, steps should
be added or replaced to achieve desired result.
8. The environment (the testing machines) does not provides hardware
resources for the automated test.
If you would like more detailed description, please let me know.
Pavlin