problems with QA:Testcase_base_startup

[Adam Williamson]

On Mon, 2014-09-29 at 20:29 -0700, Adam Williamson wrote:
> On Thu, 2014-09-18 at 08:31 -0600, Mike Ruckman wrote:
> > On Thu, Sep 18, 2014 at 06:42:07AM -0400, Kamil Paral wrote:
> > > In the base installation matrix
> > > https://fedoraproject.org/wiki/Test_Results:Fedora_21_Alpha_RC1_Base#Test_Matrix
> > > 
> > > we have this test case:
> > > https://fedoraproject.org/wiki/QA:Testcase_base_startup
> > > 
> > > It it is marked as "Alpha" in the matrix. However, it depends on disk encryption, which is a Beta requirement. That is a problem, if disk encryption doesn't work, you can't test the rest of the test case. That means you can test Alpha criteria because you are stuck on a broken Beta feature.
> > > 
> > > Furthermore, the test case basically duplicates https://fedoraproject.org/wiki/QA:Testcase_partitioning_guided_encrypted , from my POV.
> > > 
> > > So I suggest to amend Testcase_base_startup and change this sentence:
> > > "Follow all defaults, with the exception that you should encrypt a system partition (e.g. the root partition)."
> > > to this:
> > > "Follow all defaults, with the exception that you may encrypt a system partition (e.g. the root partition)."
> > > 
> > > And of course the expected results would be a slightly differently formulated as well.
> > > 
> > > This will fix the discrepancy (Alpha test case depending on Beta features), and remove the duplication.
> > > Thoughts?
> > 
> > I agree with John. Yanking the encryption bit completely makes good sense.
> 
> So, I think something's gotten lost somewhere along the way, here. My
> recollection is that Base is supposed to test keymaps, but it doesn't
> actually seem to, so indeed it seems to be a dupe of the encryption test
> case so far as encryption goes.
> 
> I agree this nexus seems rather messy. What we need to test, in broad
> terms, is:
> 
> If you install and boot either graphically or non-graphically with
> encryption and a non-US keymap, everything should work.
> 
> The Base test case in particular does read a bit like a hot mess with
> its attempt to cover both graphical and non-graphical cases. Perhaps it
> would be best to split it in two.

Well, I just revised it, and took a different approach.

I ditched the encryption stuff, and also ditched the 'run the test
twice' stuff. It seems to me it's not necessary now, as the Server
product will test the non-graphical boot path. Arguably we lose testing
that a text mode *installation* produces a bootable system, but I just
did a quick patch for that by editing
QA:Testcase_Anaconda_User_Interface_Text to recommend running
post-install tests.

I tweaked it a bit in other ways to cover our current deployment methods
better (the wording now considers disk image deployment scenarios).

I'm going through various test cases at present doing little tweaks
along these lines.

I think QA:Testcase partitioning guided encrypted handles the keymap
stuff as written.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net