Future of Test Cases and their Management in Fedora QA

[Tim Flink]

This is a conversation which has started a few times but not in anything close to a formal
way. Let's fix that so we at least have an answer of why we're doing what
we're doing :)

A test case management system (TCMS) is pretty much what it sounds like - a system which
manages at least test cases for an organization. In general, a TCMS is going to also
manage test plans and many also keep track of results. Fedora doesn't have a
traditional TCMS - we use the wiki to store our test cases and the test matrices which
roughly align to test cases, test plans and results.

The basic question is: "should we be using a more traditional TCMS instead of the
wiki?". There aren't a ton of alternatives available to us, so the comparison
list is pretty short. As far as I know, our options are:

  1. Start using Kiwi TCMS
  2. Create, deploy and maintain our own custom TCMS
  3. Keep using the wiki for our TCMS

If anyone is aware of other options which don't sacrifice the functionality we use for
release validation, are open source and aren't abandoned upstream, please reply to
this thread. Just because these 3 are the only options I could find doesn't mean that
there aren't others.

I've written up a brief summary of what I see of the advantages and disadvantages to
the three approaches to start the conversation. What are peoples' thoughts on our
options going forward? Should we look at moving on from our existing wiki-based TCMS? If
so, do either of the alternatives appear promising enough to move forward with?

Thanks in advance for everyone's time and thoughts,

Tim
  


================================================================================
1. Start using Kiwi TCMS
================================================================================

Advantages:
  - Has an existing user base, we wouldn't have to create/maintain an entire TCMS
alone
  - Uses a more structured format which can be easier to analyze
  - Allows multiple people to report results at the same time

Disadvantages:
  - Has strict relationships between products, test cases and test plans which don't
align well to how Fedora QA currently does things
  - Is currently missing some vital features (FAS integration)
  - Would require us to deploy and maintain an instance
  - Would likely require some custom patches/plutins that we'd have to keep working
with any changes in upstream KiwiTCMS
  - Does not support testcases in multiple languages

KiwiTCMS (https://kiwitcms.org/) is an open source (GPL2) TCMS which started life inside
Red Hat managing test cases for RHEL. As far as I know, it's the only open source TCMS
out there which is still maintained and is even close to our workflow. I have set up a
demo instance for people to poke at, details are in a separate email to this list.

Kiwi is written in Django so that wouldn't be a huge hurdle for our currently
available development skills but it is currently missing at least one feature before we
could use it in production (FAS authentication). That being said, Kiwi is rather
opinionated on what the relationship between testcases, test plans, builds and results
should look like. We would have to make some non-trivial changes to how we do things like
release validation and that's a non-trivial cost.


================================================================================
2. Create, deploy and maintain our own custom TCMS
================================================================================

Advantages:
  - Could be whatever we wanted it to be
  - Would integrate well with existing Fedora systems

Disadvantages:
  - Writing our own, custom TCMS would be expensive and take time
  - We'd have another system to maintain

There's always the option of writing our own TCMS. While it'd be great to have
something that uses ResultsDB out of the box and works perfectly with our workflows, that
would come at a cost. The question comes down to whether we'd gain enough from a
custom TCMS to justify the work we'd have to put into the system.


================================================================================
3. Continue Using WikiTCMS
================================================================================

Advantages:
  - Doesn't require any changes to how we're doing things
  - Is very flexible - can do pretty much whatever we want to without a ton of code
  - Doesn't require much maintenance or custom code beyond wikitcms

Disadvantages:
  - More difficult to analyze data contained in the wiki
  - Only one person can report results in a given matrix at a time
  - QA is the only group still using the Fedora wiki

This kinda started a while back when CPE started asking about whether we still needed the
wiki or not. The current Fedora wiki is a source of frustration for them due to the amount
of spam and bot issues that they have to deal with. Seeing as how QA is one of the only
(if not the only) groups left using the wiki, it's worth asking whether we really
still need it or not. It does sound like most of CPE's concerns could be satisfied
with a new wiki instance which could be locked down to only allow pages with a certain
format but I don't believe that conversation has even been started with them.

A lot of what the wiki has going for it is quite simply "if it ain't broke,
don't fix it". Is the wiki an ideal TCMS? No. Does it serve our needs for the
moment? Yes. There are so many things that we could be putting development time into,
I'm not convinced that it's worth the time and effort needed to set up a new TCMS
and change our processes to work with that new system just to say we're using a
"proper" TCMS.

[Matthew Miller]

On Wed, Jun 22, 2022 at 08:57:04AM -0600, Tim Flink wrote:
> ================================================================================
>  3. Continue Using WikiTCMS
>  ================================================================================
>  
>  Advantages:
>   - Doesn't require any changes to how we're doing things
>   - Is very flexible - can do pretty much whatever we want to without a ton of code
>   - Doesn't require much maintenance or custom code beyond wikitcms
>  
>  Disadvantages:
>   - More difficult to analyze data contained in the wiki
>   - Only one person can report results in a given matrix at a time
>   - QA is the only group still using the Fedora wiki 
From a user perspective, I discovered the QA test matrix pages long before I
realized that they weren't actually meant to be used as wiki pages, and that
they were actually managed by a tool. It's easy to mess things up. And the
tool is somewhat klunky if you're not used to it.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Tim Flink]

On 6/22/22 09:55, Matthew Miller wrote:
>  On Wed, Jun 22, 2022 at 08:57:04AM -0600, Tim Flink wrote:
> > ================================================================================
> > 3. Continue Using WikiTCMS
> > ================================================================================
> >
> > Advantages:
> >   - Doesn't require any changes to how we're doing things
> >   - Is very flexible - can do pretty much whatever we want to without a ton of code
> >   - Doesn't require much maintenance or custom code beyond wikitcms
> >
> > Disadvantages:
> >   - More difficult to analyze data contained in the wiki
> >   - Only one person can report results in a given matrix at a time
> >   - QA is the only group still using the Fedora wiki
>  
>   From a user perspective, I discovered the QA test matrix pages long before I
>  realized that they weren't actually meant to be used as wiki pages, and that
>  they were actually managed by a tool. It's easy to mess things up. And the
>  tool is somewhat klunky if you're not used to it. 
That's a good point. I guess I've been using the wiki matrices for so long that I
don't think about it much anymore.

Do we have any reasonable way of estimating how many testers/results we lose because of
the current interface? I suspect that anything we end up using is going to have a learning
curve for new people but in this case, I'm not really sure which is the best/worst.

Tim

[Adam Williamson]

On Wed, 2022-06-22 at 11:55 -0400, Matthew Miller wrote:
>  On Wed, Jun 22, 2022 at 08:57:04AM -0600, Tim Flink wrote:
>  > ================================================================================
>  > 3. Continue Using WikiTCMS
>  > ================================================================================
>  > 
>  > Advantages:
>  >  - Doesn't require any changes to how we're doing things
>  >  - Is very flexible - can do pretty much whatever we want to without a ton of code
>  >  - Doesn't require much maintenance or custom code beyond wikitcms
>  > 
>  > Disadvantages:
>  >  - More difficult to analyze data contained in the wiki
>  >  - Only one person can report results in a given matrix at a time
>  >  - QA is the only group still using the Fedora wiki
>  
>  From a user perspective, I discovered the QA test matrix pages long before I
>  realized that they weren't actually meant to be used as wiki pages, and that
>  they were actually managed by a tool. It's easy to mess things up. And the
>  tool is somewhat klunky if you're not used to it. 
Well, hand editing them is still Officially Supported(tm). You're just
supposed to *not* screw it up. If you do screw it up, one of two things
will happen. If one of the tools that uses wikitcms noticeably chokes
on it, I'll probably notice at some point, grumble a bit, and fix it.
If I'm feeling super enthusiastic I might fix wikitcms to handle
whatever you did wrong (it handles quite a lot of common human
errors/inconsistencies). If nothing that uses wikitcms chokes
*obviously* on it, we might never notice and it'll just be there
forever!

I try to steer people to using `relval report-results` because it
should be *somewhat* easier than hand-editing the pages, and it should
not ever write things in a way wikitcms doesn't parse properly. But you
are "allowed" to edit the pages by hand, too. (I actually do this
myself quite a lot, still). Just please don't screw it up. :P

There are ways to improve on this that *don't* involve switching tools,
of course. One option would be to enhance the testdays webapp quite a
bit and use it for this too. We could also make `relval report-results`
better, write a more graphical/web-y version of it, or something. I
have never put *that* much work into it, honestly. It's not that far
evolved beyond "just enough to do the job for the guy who wrote it". If
someone felt inclined, you could improve its interface quite a bit.

If we got either of those approaches far enough along, we could
consider locking out direct editing of the wiki pages somehow, if
that's plausible in mediawiki, and only allowing tools/bots to edit
them.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Adam Williamson]

On Wed, 2022-06-22 at 08:57 -0600, Tim Flink wrote:


> ================================================================================
>  3. Continue Using WikiTCMS
>  ================================================================================
>  
>  Advantages:
>    - Doesn't require any changes to how we're doing things
>    - Is very flexible - can do pretty much whatever we want to without a ton of code
>    - Doesn't require much maintenance or custom code beyond wikitcms 
Another advantage of the current system that I'd like to highlight is
that it allows us to synthesize manual and automated test results. That
is, in the current wiki results pages, you see the results from openQA
and human testers together.

I would definitely want any replacement for it to preserve this
advantage - i.e., I would want it to be able to track results from both
human testers and automated systems. Ideally, this would be achieved by
resultsdb integration - it's a bit difficult to imagine technically how
this could work, but it would be nice for results from both human
testers and automated systems to be in resultsdb, and for any human-
readable representation of the validation state to pull both sets of
results from resultsdb.

wikitcms gives us the integration of the two types of result, but it
does not currently forward results from human testers to resultsdb. So
we only have the results from automated systems in resultsdb. openQA
reports its results to *both* resultsdb and the wiki, so the wiki is
the only canonical store of *both* human and bot results.

I've toyed in the past with the idea of having `relval report-results`
also submit the result to resultsdb, but it requires solving
authentication and we still wouldn't capture human test results
submitted by editing the wiki directly. Another option would be a
wikitcms-based bot which monitored result pages for edits and forwarded
any 'new results' to resultsdb...

>  
>  Disadvantages:
>    - More difficult to analyze data contained in the wiki
>    - Only one person can report results in a given matrix at a time
>    - QA is the only group still using the Fedora wiki
>  
>  This kinda started a while back when CPE started asking about whether
>  we still needed the wiki or not. The current Fedora wiki is a source
>  of frustration for them due to the amount of spam and bot issues that
>  they have to deal with. Seeing as how QA is one of the only (if not
>  the only) groups left using the wiki, it's worth asking whether we
>  really still need it or not. It does sound like most of CPE's
>  concerns could be satisfied with a new wiki instance which could be
>  locked down to only allow pages with a certain format but I don't
>  believe that conversation has even been started with them. 
If they do get to the point where they really, really want to turn the
wiki off, I'm willing to try and handle this (setting up a reduced wiki
instance specifically for this purpose, with restrictions to allowable
page names and so on).
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Tim Flink]

On 6/22/22 14:13, Adam Williamson wrote:
>  On Wed, 2022-06-22 at 08:57 -0600, Tim Flink wrote:
>  
>  
> > ================================================================================
> > 3. Continue Using WikiTCMS
> > ================================================================================
> >
> > Advantages:
> >    - Doesn't require any changes to how we're doing things
> >    - Is very flexible - can do pretty much whatever we want to without a ton of code
> >    - Doesn't require much maintenance or custom code beyond wikitcms
>  
>  Another advantage of the current system that I'd like to highlight is
>  that it allows us to synthesize manual and automated test results. That
>  is, in the current wiki results pages, you see the results from openQA
>  and human testers together.
>  
>  I would definitely want any replacement for it to preserve this
>  advantage - i.e., I would want it to be able to track results from both
>  human testers and automated systems. Ideally, this would be achieved by
>  resultsdb integration - it's a bit difficult to imagine technically how
>  this could work, but it would be nice for results from both human
>  testers and automated systems to be in resultsdb, and for any human-
>  readable representation of the validation state to pull both sets of
>  results from resultsdb. 
For what it's worth, Kiwi does have a pretty comprehensive API when it comes to stuff
like reading or posting test results.

When it comes to storing results for automated vs. manual testing, Kiwi does seem to focus
more on having test cases which are marked as automated rather than having users marked as
bots. As far as I know, there is no way to mark a Kiwi user as a bot.

Personally, I think there are arguments for both approaches but changing over to Kiwi
would likely require a change to how we've been doing things.

>  wikitcms gives us the integration of the two types of result, but it
>  does not currently forward results from human testers to resultsdb. So
>  we only have the results from automated systems in resultsdb. openQA
>  reports its results to *both* resultsdb and the wiki, so the wiki is
>  the only canonical store of *both* human and bot results.
>  
>  I've toyed in the past with the idea of having `relval report-results`
>  also submit the result to resultsdb, but it requires solving
>  authentication and we still wouldn't capture human test results
>  submitted by editing the wiki directly. Another option would be a
>  wikitcms-based bot which monitored result pages for edits and forwarded
>  any 'new results' to resultsdb... 
At some point, we'd end up with enough chewing gum and bailing wire that we might as
well roll our own. I'm not sure we're there yet but if we start down the path of
an alternate interface to post results and/or a bot to shuffle results between the wiki
and resultsdb, I do think that would start approaching the point where we'd be better
off writing our own solution.

>> Disadvantages:
>>    - More difficult to analyze data contained in the wiki
>>    - Only one person can report results in a given matrix at a time
>>    - QA is the only group still using the Fedora wiki
>>
>> This kinda started a while back when CPE started asking about whether
>> we still needed the wiki or not. The current Fedora wiki is a source
>> of frustration for them due to the amount of spam and bot issues that
>> they have to deal with. Seeing as how QA is one of the only (if not
>> the only) groups left using the wiki, it's worth asking whether we
>> really still need it or not. It does sound like most of CPE's
>> concerns could be satisfied with a new wiki instance which could be
>> locked down to only allow pages with a certain format but I don't
>> believe that conversation has even been started with them.
> 
> If they do get to the point where they really, really want to turn the
> wiki off, I'm willing to try and handle this (setting up a reduced wiki
> instance specifically for this purpose, with restrictions to allowable
> page names and so on).

[Kamil Paral]

I have played with https://kiwidemo.fedorainfracloud.org for a few days and
I have to say I'm quite disappointed by it :-( I thought it would be closer
to what we need, with some tweaking. But currently I'm hitting serious
obstacles and the user experience for a regular tester is not great either.
It seems that Kiwi is designed in a very rigid way where each person gets
some area of responsibility and they work just there, or there's a smaller
team of educated testers which can easily collaborate on a shared area.
Other approaches feel cumbersome.

First, you need to create a test run, in order to test something. When
creating the test run, you have to define which test cases will be
included. You can populate it from a test plan, which we can prepare, but
either we'll have many many smaller test plans with a couple of test cases,
or a couple of large test plans with tens of test cases. It feels either
constraining or overwhelming to work this way. You can't just simply pick
one random test case which hasn't been tested yet, like with our wiki
matrices. Moreover, searching for those test plans is tedious, always
filling out Product, Version, etc. You're provided with a database list of
test plans, without any logical layout, milestone priorities etc, as on our
wiki. When creating a test run, you have to fill out some fields, like
Build. Perhaps we can provide pre-created hyperlinks which would forward
testers into a fully filled out page and they'd just click a button. After
you create a test run, you have to start it, and after you're done, you
have to end it. You don't actually need to, because you can populate
results even if it hasn't started/has already ended, but certain views will
display in-progress test runs and some won't (unless you display only
in-progress test runs). This feels like bookkeeping feature more than
anything else, and it's a nuisance for us. People will forget to end their
test runs.

Instead of everyone creating their own test run (actually many test runs),
it seems possible to create one single "shared" test run and distribute the
link to it. Other people can fill out the results, even if they haven't
created it. However, there are serious disadvantages:
1. Only a single result can be provided for a test case. So no longer we'd
be able to see results from multiple people, or openqa and an additional
manual tester, for one particular test case. As described above, this feels
very "corporate style of work", but very different from what we need.
2. It is extremely easy to overwrite somebody else's result by accident.
The test run page doesn't update automatically, you need to manually
refresh it. So if you simply submit a result for a test case which looks
untested, somebody else might have submitted the result already (a few
hours earlier, possibly) and your result will simply overwrite it. There's
no warning, nothing. On wiki, we at least get a warning if people edit the
same page/section simultaneously and there's an edit collision. This is a
complete no-go for us. I even wonder how corporate teams can avoid this
situation if multiple people work on the same test run, unless they're
sitting in the same office space talking to each other.

Because sharing test runs is a no-go, individual test runs is the way to
go. But there's another big hurdle. How do you figure out what should you
test, i.e. it hasn't been tested yet by somebody else? I only found a
single solution to this, which is by looking into the "Status matrix".
Unfortunately that page is probably the worst one I encountered in Kiwi.
There's no dedicated Search button, it performs a search on any field
change (and initially shows the whole database, if you wait long enough -
ooof). There's no progress indication, so you never know if a search was
initiated, if it is complete, if empty results page is actually what you
should see or not. The search params are not placed into URL, so every time
you want to refresh the page to see updated results, you have to fill out
all search fields again. The page readability is decent if only a low
number of test runs is displayed. But if people create multiple test runs
(for each smaller test plan/test section, because of submitting results
before lunch/after lunch, etc), the readability gets much worse and it's an
exercise in horizontal scrolling. Our testcase_stats (e.g. [1]) do much
better job in clearly showing what was run and what wasn't, what was the
last tested build and where were some issues. Especially if you want to
show the history, not just the current build. And of course the page again
provides just a database list of test cases, but doesn't structure them in
any way, so we can't e.g. display milestone priorities etc in that list. In
essence, the more test runs are performed, the harder it is to figure out
what wasn't run yet, any test case structuring is not possible, and you
can't refresh the page to see the current state.

The Product/Version/Build organization is usable, I think, but currently
conflicts with our way of using "environments" (different arches, bios vs
uefi, etc). Tim tried to solve it by embedding the environments into
Product, and also by populating the image names in there, but I'm not sure
it can work this way. For example, Server tests which are run just once in
general can be in the "Server" product. But if we want to distinguish
between netinst and dvd, on either uefi or bios, on three different
architectures, that would just explode the Product values. It makes
managing the test plans and seeing the current test runs results a
nightmare.

Kiwi supports some test case properties and defining applicable
environments as an experimental feature. It is not documented in their
docs, but by some trial and error I managed to define "firmware=bios,
firmware=uefi" on a test case, and then it got split up into two different
test case executions (one with bios, one with uefi) when a test run is
created. I find it a bit weird that those properties are defined in a test
case rather than in a test plan, but it would work for us, I think.
Unfortunately currently even when you can submit the results for these
"environment-defined test cases", they are not displayed in the Status
matrix. When a test case is split into two, it shows up as a single result
there, with whatever result was submitted for the first half of it. So
while this could solve some of our needs, currently it doesn't seem
finished (unsurprisingly, since it's marked experimental).

So, overall, the user experience of submitting individual test case results
is, I think, good, but everything around it (determining which test cases
haven't been tested yet, getting some guidance what to focus on, creating
and managing your test run, navigating the system, getting an overall
picture of testing) is worse than with our wiki, sometimes considerably.
(Of course I could have misunderstood many Kiwi's features, I never worked
with it before. I'll be happy to be corrected). I think that we could work
around the issues by creating:
a) a startup page, where people would see our test cases logically
separated (installation/desktop, Basic/Beta/Final, storage/package
management/etc), and just click pre-created hyperlinks leading to Kiwi
forms with all necessary input fields filled out already (assuming Kiwi
supports that). This would save people from searching in endless drop-down
menus, possibly selecting wrong things. It would also guide them on what to
do ("this section is more important than that one because it's Beta and the
other is Final").
b) an overall results page of the compose. This would help people figure
out what test cases need attention (not tested yet, marked as failed, etc)
and what the overall status of the compose is. This can be combined with
a), of course.
c) an updated testcase_stats page to see the historical progress
Pages b) and c) would pull from Kiwi and auto-generate the current picture.
The problem of environments would be solved graphically by our own pages,
and in the Kiwi internals, we'd probably do some ugly duplication or
explosion of fields like Product.

I'm honestly not sure if this is a good idea. It's a lot of work, and the
end result doesn't seem to be much better than what we currently have. Yes,
there is a structured results storage and a nice results submission page.
On the other hand, we'd need to maintain a hodge-podge of integration code
which can easily break, and yet-another-tool instance. Not to mention we
could create those part ourselves (use resultsdb and some simple html pages
for submission), if we wanted to invest time into it. Speaking as someone
who really dislikes the NIH syndrome and tries to use existing projects as
much as possible, it seems we'd gut Kiwi too much to make it worth it. I
was hoping we would be able to use it with just some slight tweaks, but so
far, from what I've seen, it doesn't seem that way.

I'll be happy if more people spend some time in it and post their
thoughts/impressions.

Kamil

[1] https://openqa.fedoraproject.org/testcase_stats/36/Installation.html