Raising the bar

[Christopher Beland]

On Mon, 2009-06-29 at 15:27 -0400, Matthias Clasen wrote:
> If you have ideas for
> other areas that could benefit from this kind of attention, please let
> us know. 

I can think of a number of different cross-component tests...

* Audio Applications - Does the sound hardware work out of the box?
  Do all applications that produce audio output work out of the box?
  (For example, I had trouble in Fedora 9 with Rosegarden, which needs
  the Jack audio server set up in addition to PulseAudio, so it
  doesn't work out of the box.)  Does it work if you try to use the
  application while another application is already producing audio
  output?  Does volume control work?  See:

https://fedoraproject.org/wiki/Bugs/Common#PulseAudio-based_volume_control_cannot_adjust_volume_satisfactorily
* Network Audio - Is it easy to play music on one computer and listen
  to it from the speakers on a different computer on the same subnet?
* File Formats - For each given file type in the known universe (.txt,
  .doc, .jpg, .ogg, .mp3, .iso, etc.) are they handled properly by
  each application you would expect to do so?  Files not handled
  internally should be passed off to the right application by all web
  browsers, IRC/instant message clients, Nautilus, etc.  All of these
  applications should use the same universal list of handlers.  If the
  necessary handler is not installed, one of these new features should
  be triggered:
   http://fedoraproject.org/wiki/Releases/FeatureCodecBuddy
   https://fedoraproject.org/wiki/Features/AutoFontsAndMimeInstaller
* Help - Does activating the help system in each application work?
  Does it actually provide helpful information?  Is documentation for
  non-network-oriented applications accessible even without a network
  connection?
* Application Capabilities - Are applications honest about what file
  types they can open (e.g. when you select File -> Open from the
  menu, is there a filter which shows you only openable files)?  If a
  given application can handle a given type of file, does that show up
  in the list of handlers in other applications (like a web browser)?
  If you open a file from within an application which it cannot
  handle, does something sensible happen (as opposed to crashing,
  getting gibberish, etc.)?  Do all applications handle non-ASCII
  filenames properly?
* Cut and Paste - Does it work smoothly everywhere, whether using
  mouse or keyboard shortcuts?  Even in the To: and Cc: fields when
  composing an email?  Even in the URL bar?  In gnome-terminal?  Does
  complex data (e.g. formatted text, graphics, multi-line text in
  single-line input boxes) get transferred properly?  Most
  importantly, does cut-and-paste handle non-ASCII text encodings
  properly (without corrupting characters)?
* International Content - Inside documents and in their own menus and
  GUI, applications should generally be able to render glyphs in any
  language or Unicode symbol set.  If a suitable font is not
  installed, this should kick in:
    https://fedoraproject.org/wiki/Features/AutoFontsAndMimeInstaller
  In some cases, suitable free fonts do not yet exist, but these cases
  are generally quite exotic.  A complex example:
    https://bugzilla.redhat.com/show_bug.cgi?id=455510
    (Undisplayable glyphs on Wikipedia)
* Menus - When someone unfamiliar with the layout thinks, "I want to
  do ___", is it easy to navigate to that function in the Gnome menus?
  If you install Everything, are all programs filed in the right
  place?  Do all programs have a suitable menu icon?  Is menu
  placement consistent with Add/Remove groups (e.g. Sound & Video vs
  Multimedia)?
* Small Screens - Is everything still usable even when the screen
  resolution is 800x600 or 648x480?  Most importantly, no
  functionality should be lost.  One good test is to open each
  application and go to Edit -> Preferences; applications with too
  many preferences will often produce a window that is not scrollable
  even though it is too big to fit on the screen.  Less important but
  good to test, is whether merely ugly or annoying things happen (like
  an application starting up with a window that is larger than the
  screen) which could be handled more gracefully.  Is there a Fedora
  desktop configuration suitable for PDAs or other small mobile
  devices?
* Press All the Buttons - Do all the keys you never use (Print Screen,
  Pause/Break, Windows, Menu, Play/Stop/Fast Forward, web shortcuts,
  etc.)  actually do something sensible, both on the desktop and in
  the context of the command line and any given application
  (e.g. watching a video).  Handy links to descriptions of traditional
  behavior:
   http://en.wikipedia.org/wiki/Keyboard_(computing)
   http://en.wikipedia.org/wiki/Category:Computer_keys
* Persistence - Do your desktop settings stay in place after you
  logout?  After you reboot?  What about numlock, capslock, etc?
* Desktop Stress Test - Add lots of applets to a panel.  Do sensible
  things happen when notification icons appear?  Do all the applets
  work as advertised?  Are any of them redundant?  Open multiple
  documents in multiple applications (OpenOffice, Firefox, Emacs,
  Gimp, etc.).  Does the default configuration do sensible things
  (it's easy to find the document you are looking for using the window
  list)?  What about alternate window-navigation tools?  What about if
  you add multiple workspaces to the mix?
* Working Remotely - Do applications function properly if you are
  running them remotely via ssh or other tool?  Some known problems
  are related to:
   https://bugzilla.redhat.com/show_bug.cgi?id=484171
   (ssh doesn't forward dbus)
  Applications should function acceptably even given this limitation.
* Crashes - All crashes should bring up a user-friendly assistant to
  report the problem.  Bug reports should be successfully filed in a
  suitable bug tracker without any special knowledge on the part of
  the reporter.  Crashes can be induced from the command line with
  "kill -SEGV".  More info:
   http://fedoraproject.org/wiki/Features/ABRT
  (Some applications use different crash-handling software.)
  It would also be nice if applications recovered from crashes without
  significant data loss.  (Can be tested by opening a document and
  making several changes without saving, waiting a little while to let
  any periodic auto-save kick in, then inducing a crash.)  Obviously
  any real crashes that are encountered during testing should be
  reported, since they shouldn't happen in the first place.

As testing days are finished, it might be interesting to output a
not-very-detailed but growing list similar to
http://fedoraproject.org/wiki/QA/ReleaseCriteria but for "things the
user interface should do seamlessly".  As in, "if it's not doing this
for you, you should file a bug" and also as guide for future re-testing.
Or maybe it's more efficient to leave tracking up to Bugzilla and just
trust people to know something is wrong (whether it's a bug or a design
flaw) when they see it.

-B.