In case some people on this list missed the thread on Fedora-Devel, there are some rather nice ideas (or not :) for Bodhi below.
Original email: http://lists.fedoraproject.org/pipermail/devel/2010-March/131723.html
---------- Forwarded message ---------- From: Aaron Faanes dafrito@gmail.com Date: Mon, Mar 1, 2010 at 15:22 Subject: Fight bugs, not FESCo To: devel@lists.fedoraproject.org
<snip>
Look at it this way: In a perfect scenario, every user of a package runs every test the moment it hits updates-testing. That means that a package only needs to stay in updates-testing for however long the package tests take. Barely any time at all! We don't live in an ideal world, though, but I think the formula looks something like this:
Quality of Product = Quality of Testing Quality of Testing = Test Coverage * Number of Configurations
Configurations are your users, and coverage is depth. Coverage is also accuracy: If I spend 30 minutes testing every preference even though you didn't change any preferences, then I just wasted my time on your package. The goal is to make tests easy and specific for the tester, and you'll find more testers and better test coverage.
* Increase coverage through Bodhi
As a tester, I love Bodhi. It shows me new things to test. If I have a list of bugs that were fixed in this build, they're all test-cases. If I have a changelog that shows features that were added, each change is a test-case. Unfortunately, if I have neither, then I can only really do a smoke test. Here's a few concrete ideas to encourage better coverage through Bodhi:
- Allow test cases to be added to Bodhi. These already exist for the RCs, they could optionally be written by maintainers too. It doesn't have to be formal, though. "Test the printing stuff" is sufficient. - Integrate changelogs where available. Since these show what changed, testers can use them to be more efficient and thorough with their tests, increasing coverage. - Further integrate Bugzilla and Bodhi. If I'm able to see what bugs are open for a package through Bodhi, I can check if those bugs were affected by the build I'm testing. I could also see if a bug is already open for this build without having to search through Bugzilla, increasing my efficiency. - Tweak karma. I feel bad leaving negative karma and I shouldn't! Bug reports are healthy. Also, positive karma is rare and subjective. I think they should either be plain comments, or a per-user checkbox for a given test-case.
Test cases don't have to be formal. We want maintainers and testers to use these features, not hate them. They shouldn't get in the way. But, if a maintainer wants users to test an area, it should be easy for him to let users know and focus their feedback towards that area. Empower maintainers and testers to figure out what works best.
<snip>
-- Aaron Faanes --
---------- Mathieu Bridon