In my quest for reducing the number of test cases, I intend to merge following test cases:
https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_VNC https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_VNC_Vncco... https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_VNC_Passw...
The reason is that currently we require 6 acts to test VNC (3 test cases for two architectures). I believe the test cases are really similar (if one is broken, there is a high probability the others are broken too) and the time+energy requirements are too high for this one installer feature.
In the perfect world, we have 10^30 test cases for each available path in the installer and we can test them all. We're not in a perfect world. I'd like to decrease the number of test cases that are not vital by making them fuzzy - there are several paths the tester can choose. We won't know exactly which path the tester chose (unless he finds and reports a bug), but it is likely that different people will choose slightly different paths (or one person redoing the test case will probably choose a slightly different path next time).
This approach lowers the time+energy requirements of "less important" test cases, at the expense of rigid testing (rigid testing is good for coverage, but bad for performance). This way we can spend the gained resources at other "more important" tasks and test cases (e.g. partitioning code is 100x more often broken then VNC, so it's good to spend less time on VNC and more on partitioning).
As a result, there would be a single test case for VNC, that would ask people to either connect to anaconda's VNC server (with optional password), or ask anaconda to connect to your running VNC server (a.k.a vncconnect).
Any concerns or comments?
On Wed, 2012-11-07 at 09:36 -0500, Kamil Paral wrote:
In my quest for reducing the number of test cases, I intend to merge following test cases:
https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_VNC https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_VNC_Vncco... https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_VNC_Passw...
The reason is that currently we require 6 acts to test VNC (3 test cases for two architectures). I believe the test cases are really similar (if one is broken, there is a high probability the others are broken too) and the time+energy requirements are too high for this one installer feature.
In the perfect world, we have 10^30 test cases for each available path in the installer and we can test them all. We're not in a perfect world. I'd like to decrease the number of test cases that are not vital by making them fuzzy - there are several paths the tester can choose. We won't know exactly which path the tester chose (unless he finds and reports a bug), but it is likely that different people will choose slightly different paths (or one person redoing the test case will probably choose a slightly different path next time).
This approach lowers the time+energy requirements of "less important" test cases, at the expense of rigid testing (rigid testing is good for coverage, but bad for performance). This way we can spend the gained resources at other "more important" tasks and test cases (e.g. partitioning code is 100x more often broken then VNC, so it's good to spend less time on VNC and more on partitioning).
As a result, there would be a single test case for VNC, that would ask people to either connect to anaconda's VNC server (with optional password), or ask anaconda to connect to your running VNC server (a.k.a vncconnect).
Any concerns or comments?
I think that's going too far. The VNC and VNC_Vncconnect cases are very different codepaths.
I think it'd be okay to merge VNC and VNC_Password and just require the testing to include use of a password. But Vncconnect should remain separate. We could consider dropping it from being a blocking test, though. I'm not sure we actually need to block release on the vnc connect functionality, though there may be some reason to. Is anyone aware of the use cases for the vncconnect functionality and how critical they are?
I think that's going too far. The VNC and VNC_Vncconnect cases are very different codepaths.
I think it'd be okay to merge VNC and VNC_Password and just require the testing to include use of a password.
OK, at least merging these two seems like a no-brainer to me. I'll leave the password optional, because that way both options get tested eventually.
But Vncconnect should remain separate. We could consider dropping it from being a blocking test, though. I'm not sure we actually need to block release on the vnc connect functionality, though there may be some reason to. Is anyone aware of the use cases for the vncconnect functionality and how critical they are?
I think the main use case is if you want to install a machine behind a firewall, and you have a public IP. Whether that is critical enough to block the release or not, I do not know.
Anyone else has an opinion on that?
I think that's going too far. The VNC and VNC_Vncconnect cases are very different codepaths.
I think it'd be okay to merge VNC and VNC_Password and just require the testing to include use of a password.
OK, at least merging these two seems like a no-brainer to me. I'll leave the password optional, because that way both options get tested eventually.
I have merged the two test cases into one: https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_VNC
But Vncconnect should remain separate. We could consider dropping it from being a blocking test, though. I'm not sure we actually need to block release on the vnc connect functionality, though there may be some reason to. Is anyone aware of the use cases for the vncconnect functionality and how critical they are?
I think the main use case is if you want to install a machine behind a firewall, and you have a public IP. Whether that is critical enough to block the release or not, I do not know.
Anyone else has an opinion on that?
If there are no further comments, I'll probably leave it as it is, for the time being.