On Sun, 2016-07-10 at 21:30 +0530, Sayan Chowdhury wrote:
> Hi,
>
> I recently packaged and pushed an update for fedmsg-meta-fedora-infrastructure
> to bodhi and exactly 40 secs[1] later I got a +1 to the update. I am sure that
> testing a package surely takes more than 40 secs. This makes me really curious
> that are the packages really being tested before giving out the karma.
>
> After going through messages in datagrepper[2][3], I found that few people are
> giving out karma in one go (4-5 packages under a minute). If these packages
> really are not-tested and the karma are given out randomly then I am sure that
> this sure going to affect the release, infrastructure and our users.
>
> Does anybody know what is going on?

So I've been discussing this with various people in the last few days,
and one specific idea has come out of that which I'd like to float.

We've been hesitant to suggest this before as we thought packagers
might not like the idea, but we figured it can't hurt to at least
suggest it.

The idea is this: there could be a requirement for all packages to
provide at least *some* kind of 'how to test' information.

Looked at from the perspective of a new tester, the current system is
quite difficult to handle when it comes to upgrades of packages which
aren't obviously part of the critical path (e.g. kernel) or a well-
known GUI application package (e.g. firefox).

How do we *expect* a new tester to respond when they come to an update
for fedmsg-meta-fedora-infrastructure , in Bodhi or fedora-easy-karma?
It's very difficult - probably impossible - for them to know or work
out what they should actually do to test this package.

Of course, writing instructions for every single package is a lot of
work, but right now we have test cases for very few packages, and it
would definitely help if we could significantly increase that number.

What do people think about this idea?

To be clear, the idea would be to have general-purpose instructions for
basic functionality testing of each package, not requiring new 'how to
test' text to be written for every individual package update,
specifically tailored to the changes in that update.

The way the system works at present is that Bodhi will show all wiki
pages in a specific category based on the package name. For package
'foo', Bodhi's web interface and fedora-easy-karma will list all wiki
pages in the category "Category:Package_foo_test_cases" . These are
expected to be typical 'test case' pages, though there isn't actually
any technical *requirement* for them to be, or enforcement of that.
--

Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
--
devel mailing list
devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org