On Wed, 2012-11-07 at 09:36 -0500, Kamil Paral wrote:
In my quest for reducing the number of test cases, I intend to merge following test cases:
https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_VNC https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_VNC_Vncco... https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_VNC_Passw...
The reason is that currently we require 6 acts to test VNC (3 test cases for two architectures). I believe the test cases are really similar (if one is broken, there is a high probability the others are broken too) and the time+energy requirements are too high for this one installer feature.
In the perfect world, we have 10^30 test cases for each available path in the installer and we can test them all. We're not in a perfect world. I'd like to decrease the number of test cases that are not vital by making them fuzzy - there are several paths the tester can choose. We won't know exactly which path the tester chose (unless he finds and reports a bug), but it is likely that different people will choose slightly different paths (or one person redoing the test case will probably choose a slightly different path next time).
This approach lowers the time+energy requirements of "less important" test cases, at the expense of rigid testing (rigid testing is good for coverage, but bad for performance). This way we can spend the gained resources at other "more important" tasks and test cases (e.g. partitioning code is 100x more often broken then VNC, so it's good to spend less time on VNC and more on partitioning).
As a result, there would be a single test case for VNC, that would ask people to either connect to anaconda's VNC server (with optional password), or ask anaconda to connect to your running VNC server (a.k.a vncconnect).
Any concerns or comments?
I think that's going too far. The VNC and VNC_Vncconnect cases are very different codepaths.
I think it'd be okay to merge VNC and VNC_Password and just require the testing to include use of a password. But Vncconnect should remain separate. We could consider dropping it from being a blocking test, though. I'm not sure we actually need to block release on the vnc connect functionality, though there may be some reason to. Is anyone aware of the use cases for the vncconnect functionality and how critical they are?