This is a conversation which has started a few times but not in anything close to a formal
way. Let's fix that so we at least have an answer of why we're doing what
we're doing :)
A test case management system (TCMS) is pretty much what it sounds like - a system which
manages at least test cases for an organization. In general, a TCMS is going to also
manage test plans and many also keep track of results. Fedora doesn't have a
traditional TCMS - we use the wiki to store our test cases and the test matrices which
roughly align to test cases, test plans and results.
The basic question is: "should we be using a more traditional TCMS instead of the
wiki?". There aren't a ton of alternatives available to us, so the comparison
list is pretty short. As far as I know, our options are:
1. Start using Kiwi TCMS
2. Create, deploy and maintain our own custom TCMS
3. Keep using the wiki for our TCMS
If anyone is aware of other options which don't sacrifice the functionality we use for
release validation, are open source and aren't abandoned upstream, please reply to
this thread. Just because these 3 are the only options I could find doesn't mean that
there aren't others.
I've written up a brief summary of what I see of the advantages and disadvantages to
the three approaches to start the conversation. What are peoples' thoughts on our
options going forward? Should we look at moving on from our existing wiki-based TCMS? If
so, do either of the alternatives appear promising enough to move forward with?
Thanks in advance for everyone's time and thoughts,
Tim
================================================================================
1. Start using Kiwi TCMS
================================================================================
Advantages:
- Has an existing user base, we wouldn't have to create/maintain an entire TCMS
alone
- Uses a more structured format which can be easier to analyze
- Allows multiple people to report results at the same time
Disadvantages:
- Has strict relationships between products, test cases and test plans which don't
align well to how Fedora QA currently does things
- Is currently missing some vital features (FAS integration)
- Would require us to deploy and maintain an instance
- Would likely require some custom patches/plutins that we'd have to keep working
with any changes in upstream KiwiTCMS
- Does not support testcases in multiple languages
KiwiTCMS (
https://kiwitcms.org/) is an open source (GPL2) TCMS which started life inside
Red Hat managing test cases for RHEL. As far as I know, it's the only open source TCMS
out there which is still maintained and is even close to our workflow. I have set up a
demo instance for people to poke at, details are in a separate email to this list.
Kiwi is written in Django so that wouldn't be a huge hurdle for our currently
available development skills but it is currently missing at least one feature before we
could use it in production (FAS authentication). That being said, Kiwi is rather
opinionated on what the relationship between testcases, test plans, builds and results
should look like. We would have to make some non-trivial changes to how we do things like
release validation and that's a non-trivial cost.
================================================================================
2. Create, deploy and maintain our own custom TCMS
================================================================================
Advantages:
- Could be whatever we wanted it to be
- Would integrate well with existing Fedora systems
Disadvantages:
- Writing our own, custom TCMS would be expensive and take time
- We'd have another system to maintain
There's always the option of writing our own TCMS. While it'd be great to have
something that uses ResultsDB out of the box and works perfectly with our workflows, that
would come at a cost. The question comes down to whether we'd gain enough from a
custom TCMS to justify the work we'd have to put into the system.
================================================================================
3. Continue Using WikiTCMS
================================================================================
Advantages:
- Doesn't require any changes to how we're doing things
- Is very flexible - can do pretty much whatever we want to without a ton of code
- Doesn't require much maintenance or custom code beyond wikitcms
Disadvantages:
- More difficult to analyze data contained in the wiki
- Only one person can report results in a given matrix at a time
- QA is the only group still using the Fedora wiki
This kinda started a while back when CPE started asking about whether we still needed the
wiki or not. The current Fedora wiki is a source of frustration for them due to the amount
of spam and bot issues that they have to deal with. Seeing as how QA is one of the only
(if not the only) groups left using the wiki, it's worth asking whether we really
still need it or not. It does sound like most of CPE's concerns could be satisfied
with a new wiki instance which could be locked down to only allow pages with a certain
format but I don't believe that conversation has even been started with them.
A lot of what the wiki has going for it is quite simply "if it ain't broke,
don't fix it". Is the wiki an ideal TCMS? No. Does it serve our needs for the
moment? Yes. There are so many things that we could be putting development time into,
I'm not convinced that it's worth the time and effort needed to set up a new TCMS
and change our processes to work with that new system just to say we're using a
"proper" TCMS.