"Testcase base startup" confusion

[Adam Williamson]

On Tue, 2012-12-18 at 05:26 -0500, Kamil Paral wrote:
> > The point here is that 'does the install succeed?' and 'does the
> > installed system boot?' are really separate test cases. Say you
> > install
> > an encrypted system, right? Say anaconda does everything just right,
> > but
> > there's a bug in dracut and you can't enter the passphrase on boot of
> > the installed system.
> > 
> > That's really nothing to do with installation. It's not a bug in the
> > installer. It shouldn't be reported against anaconda. So it really
> > shouldn't be part of the 'installation' test case. This was the
> > reason
> > for splitting this 'startup' test case out on its own in the first
> > place
> > - to make the distinction between 'failure during install' and
> > 'failure
> > on boot' clearer. I'm not a fan of 'install' test cases that include
> > 'check the installed system boots'.
> > 
> > I don't mind splitting it out further, at all, but I'm opposed to
> > just
> > saying 'oh, the install test case covers it', that's going backwards.
> 
> Actually I see it exactly the other way. I believe the expected result
> "the installed system initiates boot properly" should be part of
> _every_ installation test case. The reason is that you can't know
> whether the installation succeeded if you haven't tried to boot it.
> Anaconda could have reported success but failed to install grub, or
> something similar (happened just a few days ago). Or it could've
> messed up some files on /boot. If you don't check that it can enter
> grub and then load kernel+initrd and mount root filesystem, you can't
> really say the installation was successful.

> If we don't provide this instruction in our installation test cases,
> but rather provide a single "system boots" test case, then we might
> miss some important bugs where the system fails to be just in certain
> scenarios (certain installation types). People might perform 10
> installation test cases and report success immediately after seeing
> anaconda "finish" screen, without letting it boot. And then test the
> system boot just _once_. That is far from ideal. Moreover, checking
> that system boot works is not a waste of time, it's very fast.
> 
> I think what you want to see is a green color in an installation test
> case cell, and a red color in a boot test case cell, if there is a
> dracut bug, right? That would be nice, but that is not achievable,
> unless we replace the expected result "installation succeeded" with
> "anaconda reported success" (those are different things). Yes, in that
> case we can separate installation and boot. But I don't believe what
> anaconda says and I don't think we should, we should check it instead.

More or less. I don't think the problem is impossible. I recognize your
scenario, but still, there is a difference between a system that doesn't
boot because of a failure in the installer and a system that doesn't
boot for some other reason.

> Also I think the test case separation makes sense only when different
> release criteria milestones are mixed. If test case A is Alpha and
> test case B is Beta, and they are quite separate things, yes, two test
> cases. But if both are Alpha and they are very tightly related
> (testing one also tests the outcome of the second), why would we
> separate them? It violates Alpha and it can be a single red cell, no
> problem.


-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net