Fedora 21 Alpha validation test work

Fri Jan 23 22:57:02 UTC 2015

On Fri, 2014-07-11 at 13:23 -0400, Matthias Clasen wrote:
> Hey Adam,
> 
> I've looked over the workstation testcases quickly.

I just found this mail a half year later, so some very belated 
replies...

> Some comments:
> 
> https://fedoraproject.org/wiki/QA:Testcase_desktop_updates talks 
> about yum and PackageKit - should it include dnf ?

As Jaro said at the time, yum is still the default/supported package 
manager for F21. For F22 we're currently slated to change to dnf, and 
if that *happens* we'll change the test case, but there's still some 
discussion about it at present.

> https://fedoraproject.org/wiki/QA:Testcase_desktop_panel_basic could 
> benefit from some outline of what desktop 'panel' elements we expect 
> to see. Sticking with the most common laptop case, I would mention 
> network, sound and battery status as expected

So yeah, a thing to bear in mind is that most of the cases were 
written around F13 time (IIRC), with the GNOME 2 and KDE 3 of those 
days in mind, when everything looked more or less like Windows 98 and 
everyone's 'panel' actually had some bits running in it that they 
didn't write (like nm-applet was a pretty separate thing from gnome-
panel, for e.g.)

When I'm running this test (and the 'advanced' version) against Shell I
usually check all the bits of the Shell top bar, user menu, and 
overview. Re-wording it to be too specific about the expected elements 
runs the risk of no longer being applicable to the other desktops, but 
of course if it's necessary we can have two (or more) cases, or some 
conditional language.

I think this just about scrapes by as 'OK' at present, but if I can 
find the time I'll see if I can improve it for the Modern World.

> https://fedoraproject.org/wiki/QA:Testcase_desktop_login seems to be 
> a bit of a misnomer; the test is really about keyboard layouts.

Hum, not really, it really *does* test login/DM stuff - it just also 
tests that there aren't screwups with the keyboard layout not being 
consistent between installer, DM, and desktop. IIRC, I initially 
started writing a separate 'keyboard layouts' test case at some point, 
then realized that what it was actually doing was duplicating a bunch 
of existing test cases but with 'oh, and do it with a different 
keyboard layout' - so instead of doing that, I just wrote 'check it 
with a different keyboard layout' into each of the existing tests. 
There are similar bits in the disk encryption test case, for e.g., to 
make sure the right keyboard layout is used for entering the 
passphrase on boot.

> https://fedoraproject.org/wiki/QA:Testcase_audio_basic talks about 
> selection hw profiles. I really think that the expectation for 
> 'common hw' (ie your typical laptop) should be that sound works out 
> of the box.

Reasonable point, I'll have to check on the current actual status of 
this with different hardware; I *think* at least on modern systems the 
drivers/PA should be able to sense what's connected and choose the 
appropriate output profile, but I'm not 100% sure.

I think the note for multiple *devices* is still valid, though. If you 
have multiple output devices, PA can't magically tell which one you 
want sound to come out of, you do have to tell it.

> https://fedoraproject.org/wiki/QA:Testcase_desktop_updates same 
> comment as above - should this use dnf instead of yum ? I think it 
> would be good to make this testcase more specific wrt to the way 
> offline updates are integrated in the desktop now. Test that we 
> notify (once per day) about available updates. Test that the 
> logout/poweroff dialog offers to install pending updates. Test that 
> we notify about successful and unsuccessful offline updates, and 
> show details.

There's definitely some ancient text in there ("observing whether the 
'checking for updates' and then 'updates available' icons appear in 
the notification area", no, that's not going to happen). The same 
problem as I mentioned above applies to making this *too* specific to 
Workstation - the same test case is applied to KDE and other lives too 
- but I think the same approach will work, first try and 'modernize' 
it as a generic test case, and then evaluate whether it's 
necessary/worthwhile to split out a Workstation-specific case from it.

> https://fedoraproject.org/wiki/QA:Testcase_workstation_theming would 
> benefit from adding some mention of accessibility - we expect the 
> HighContrast theme to work and be of similar quality to the default 
> theme.

Thanks, I'm not sure whether that'd work best as an addon to this test 
case or a new one, but either way, I'll try and add something to cover 
it.

> Things that would be good to capture in new testcases:
> 
> - Screen locking. Explicit locking via keyboard shortcut or menu 
> should
> work, as well as lock on idle. Screen blanking after lock is 
> something that frequently has issues. Can also test that inhibiting 
> works (when watching video in totem, or in the browser.

Noted, thanks!

-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net