Testing instructions / advice in update descriptions: possibly not a good idea
I've noticed that lately at least a couple of developers have put
testing instructions for updates into the update descriptions.
It's awesome that developers are being proactive in providing testing
instructions. Really awesome. you are rock stars. THANK YOU.
Having said that, putting them in the update descriptions is possibly
not a great idea, because the update descriptions are permanent -
they're there once the updates go to stable, not just while they're in
testing. (You may be able to remove them by editing the update - I
haven't checked this - but still, that relies on you remembering to
change the update description when submitting it to stable). This means
that 'regular users' who really aren't interested in testing are going
to see the testing instructions when they're just running the daily,
easy-as-falling-off-a-log, graphical update process (PackageKit shows
the update descriptions), and they're likely to be confused / worried -
"why am I getting an update which comes with testing instructions? Is it
likely to be broken? Am I supposed to be testing it? Are my children
safe? WON'T SOMEBODY PLEASE THINK OF THE CHILDREN?!"
:)
We already had a thread on test list started by a worried user -
http://lists.fedoraproject.org/pipermail/test/2010-December/095968.html . So, while once
again it is super awesome that developers want to provide testing instructions, please
think twice about putting them in the update description field.
I've been working on a test case-based system for providing these kinds
of instructions in a trac ticket -
https://fedorahosted.org/fedora-qa/ticket/154 - and am probably going to
post a message to this list and test list today or tomorrow with a draft
framework and example test case, which will hopefully be enough for both
developers and QA folks to start writing test cases within the framework
(QA will also be happy to offer help to any developers who want to
provide instructions but aren't totally confident about how best to
frame them as a test case, but it's pretty easy, really). That should be
a better way to provide testing directions. We'll also be working with
the related tooling developers (bodhi and fedora-easy-karma in
particular) to try and integrate display of the relevant test cases in
appropriate situations, to make sure testers are aware of the
instructions.
Thanks!
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net