[Fedora QA] #86: Add test coverage for text-mode upgrade

[fedora-badges]

#86: Add test coverage for text-mode upgrade
---------------------------+------------------------------------------------
 Reporter:  jlaska         |       Owner:           
     Type:  enhancement    |      Status:  new      
 Priority:  major          |   Milestone:  Fedora 14
Component:  Wiki           |     Version:           
 Keywords:  retrospective  |  
---------------------------+------------------------------------------------
 = problem =

See [http://bugzilla.redhat.com/590823 RHBZ#590823 - Text-mode upgrade
 fails when installing new bootloader -- KeyError: 'bootloader']

= analysis =

Some tests in the
 [https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test
 install test matrix] involve user interface components in both text-mode
 and graphical-mode (native or VNC). One such bug (RHBZ #590823) involved a
 problem found only during text-mode upgrades to Fedora 13.  The text-mode
 upgrade screen shows different options thanenhancement recommendation =

= enhancement recommendation =

I'm hesitant to recommend specific text-mode tests for each user interface
 element that differs in text-mode and graphical-mode (it would be
 exhaustive but unachievable).  As a catch-all, testing a text-mode install
 is included in the current matrix.  Is there something small that can be
 done to cover the gap?

1. Perhaps adding a text-mode upgrade test case to the install matrix, or
 ...
  2. Adding variations to the existing upgrade tests
 [https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_Update_Bootloade...
 QA:Testcase_Anaconda_Upgrade_Update_Bootloader] and
 [https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_Skip_Bootloader
 QA:Testcase_Anaconda_Upgrade_Skip_Bootloader] to also test text-mode, or
 ...
  3. <insert better idea here>

[fedora-badges]

#86: Add test coverage for text-mode upgrade
--------------------------+-------------------------------------------------
  Reporter:  jlaska       |       Owner:  rhe          
      Type:  enhancement  |      Status:  assigned     
  Priority:  major        |   Milestone:  Fedora 14    
 Component:  Wiki         |     Version:               
Resolution:               |    Keywords:  retrospective
--------------------------+-------------------------------------------------
Changes (by rhe):

* owner:  => rhe
  * status:  new => assigned

Comment:

I think the above suggestion1 and 2 are both ok. Test-mode generally need
 two tests:
  * 1. A fresh install.(Already included in the matrix)
  * 2. An update install. Tester can choose any bootloader configuration
 method from the two. But since the bug was discovered only on a specific
 configuration, having two test-mode update tests can have more coverage.

[fedora-badges]

#86: Add test coverage for text-mode upgrade
--------------------------+-------------------------------------------------
  Reporter:  jlaska       |       Owner:  rhe          
      Type:  enhancement  |      Status:  assigned     
  Priority:  major        |   Milestone:  Fedora 14    
 Component:  Wiki         |     Version:               
Resolution:               |    Keywords:  retrospective
--------------------------+-------------------------------------------------
Comment (by rhe):

Two tests have been drafted for upgrading in text mode:

*
 https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_Skip_Bootloader_...

*
 https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_Update_Bootloade...

[fedora-badges]

#86: Add test coverage for text-mode upgrade
--------------------------+-------------------------------------------------
  Reporter:  jlaska       |       Owner:  rhe          
      Type:  enhancement  |      Status:  assigned     
  Priority:  major        |   Milestone:  Fedora 14    
 Component:  Wiki         |     Version:               
Resolution:               |    Keywords:  retrospective
--------------------------+-------------------------------------------------
Changes (by jlaska):

* cc: clumens (added)

Comment:

Adding clumens for some clarification on anaconda upgrade bootloader
 behavior.

Replying to [comment:2 rhe]:
> Two tests have been drafted for upgrading in text mode:

https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_Skip_Bootloader_...
>

https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_Update_Bootloade...

Nice, the tests look good.

I'm still unclear on the difference in expected results for the ''Skip
 bootloader'' and the ''update bootloader'' test case.  When a new kernel
 package is installed during the upgrade, it will run grubby (see the
 %script of the kernel rpm) and modify grub.conf.  When we choose ''Skip
 bootloader'', does that mean that anaconda '''will not''' write a new
 bootloader configuration and '''will not''' run grub-install, but
 '''will''' update the existing grub.conf when the new kernel is installed?

clumens: any thoughts?

[fedora-badges]

#86: Add test coverage for text-mode upgrade
--------------------------+-------------------------------------------------
  Reporter:  jlaska       |       Owner:  rhe          
      Type:  enhancement  |      Status:  assigned     
  Priority:  major        |   Milestone:  Fedora 14    
 Component:  Wiki         |     Version:               
Resolution:               |    Keywords:  retrospective
--------------------------+-------------------------------------------------
Comment (by clumens):
> I'm still unclear on the difference in expected results for the ''Skip

bootloader'' and the ''update bootloader'' test case.  When a new kernel
 package is installed during the upgrade, it will run grubby (see the
 %script of the kernel rpm) and modify grub.conf.  When we choose ''Skip
 bootloader'', does that mean that anaconda '''will not''' write a new
 bootloader configuration and '''will not''' run grub-install, but
 '''will''' update the existing grub.conf when the new kernel is installed?
> clumens: any thoughts?

Regardless of what option you pick on the bootloader UI, the kernel's
 %post scriptlet will still get run.  So you can't help but get grub-
 install run and grub.conf changed.

If you select "Skip bootloader", you won't get the screen that allows you
 to setup how the bootloader should be configured, anaconda won't write out
 a bootloader configuration for you, and it will not run whatever program
 installs the bootloader for your platform.  You will, however, still get
 whatever happens when the kernel's %post script runs.

Does that clear things up a bit?

[fedora-badges]

#86: Add test coverage for text-mode upgrade
--------------------------+-------------------------------------------------
  Reporter:  jlaska       |       Owner:  rhe          
      Type:  enhancement  |      Status:  assigned     
  Priority:  major        |   Milestone:  Fedora 14    
 Component:  Wiki         |     Version:               
Resolution:               |    Keywords:  retrospective
--------------------------+-------------------------------------------------
Comment (by rhe):

Replying to [comment:4 clumens]:
> > I'm still unclear on the difference in expected results for the ''Skip

bootloader'' and the ''update bootloader'' test case.  When a new kernel
 package is installed during the upgrade, it will run grubby (see the
 %script of the kernel rpm) and modify grub.conf.  When we choose ''Skip
 bootloader'', does that mean that anaconda '''will not''' write a new
 bootloader configuration and '''will not''' run grub-install, but
 '''will''' update the existing grub.conf when the new kernel is installed?
> > clumens: any thoughts?
>
> Regardless of what option you pick on the bootloader UI, the kernel's

%post scriptlet will still get run.  So you can't help but get grub-
 install run and grub.conf changed.
> If you select "Skip bootloader", you won't get the screen that allows

you to setup how the bootloader should be configured, anaconda won't write
 out a bootloader configuration for you, and it will not run whatever
 program installs the bootloader for your platform.

Thanks clumens, Then is there any way to test the difference between
 “update” and "skip" bootloader? The existing
 [https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_Update_Bootloade...
 update bootloader] test has the same results no matter one chooses update
 or skip bootloader.

[fedora-badges]

#86: Add test coverage for text-mode upgrade
--------------------------+-------------------------------------------------
  Reporter:  jlaska       |       Owner:  rhe          
      Type:  enhancement  |      Status:  assigned     
  Priority:  major        |   Milestone:  Fedora 14    
 Component:  Wiki         |     Version:               
Resolution:               |    Keywords:  retrospective
--------------------------+-------------------------------------------------
Comment (by clumens):
> Thanks clumens, Then is there any way to test the difference between

“update” and "skip" bootloader? The existing
 [https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_Update_Bootloade...
 update bootloader] test has the same results no matter one chooses update
 or skip bootloader.

I can't think of anything, no.

[fedora-badges]

#86: Add test coverage for text-mode upgrade
--------------------------+-------------------------------------------------
  Reporter:  jlaska       |       Owner:  rhe          
      Type:  enhancement  |      Status:  assigned     
  Priority:  major        |   Milestone:  Fedora 14    
 Component:  Wiki         |     Version:               
Resolution:               |    Keywords:  retrospective
--------------------------+-------------------------------------------------
Comment (by jlaska):

Replying to [comment:6 clumens]:
> I can't think of anything, no.

I reached out for additional thoughts on [https://www.redhat.com/archives
 /anaconda-devel-list/2010-September/msg00169.html anaconda-devel-list].
 Maybe that will help highlight differences between upgrade and skip ... or
 combine the two options in the UI.  Stay tuned ...

[fedora-badges]

#86: Add test coverage for text-mode upgrade
--------------------------+-------------------------------------------------
  Reporter:  jlaska       |       Owner:  rhe          
      Type:  enhancement  |      Status:  assigned     
  Priority:  major        |   Milestone:  Fedora 14    
 Component:  Wiki         |     Version:               
Resolution:               |    Keywords:  retrospective
--------------------------+-------------------------------------------------
Comment (by jlaska):

Replying to [comment:7 jlaska]:
> Replying to [comment:6 clumens]:
> > I can't think of anything, no.
>
> I reached out for additional thoughts on

[https://www.redhat.com/archives/anaconda-devel-
 list/2010-September/msg00169.html anaconda-devel-list].  Maybe that will
 help highlight differences between upgrade and skip ... or combine the two
 options in the UI.  Stay tuned ...

Chris provided a great breakdown on the differences between the 3
 bootloader upgrade options (see https://www.redhat.com/archives/anaconda-
 devel-list/2010-October/msg00036.html).  I'm inclined to think using that
 mail, we can distinguish between upgraded bootloader and skipped
 bootloader by comparing the MBR (stage#1) with /boot/grub/stage1.  If they
 are equal, it was upgraded.  If they are different, it was skipped.

The following command alone does '''not''' seem to be enough to do the
 comparison.  I suspect some fiddling may be required:
 {{{
 # dd if=/dev/vda of=/tmp/mbr bs=512 count=1
 # md5sum /tmp/mbr /boot/grub/stage1
 }}}

[fedora-badges]

#86: Add test coverage for text-mode upgrade
--------------------------+-------------------------------------------------
  Reporter:  jlaska       |       Owner:  rhe          
      Type:  enhancement  |      Status:  assigned     
  Priority:  major        |   Milestone:  Fedora 15    
 Component:  Wiki         |     Version:               
Resolution:               |    Keywords:  retrospective
--------------------------+-------------------------------------------------
Changes (by jlaska):

* milestone:  Fedora 14 => Fedora 15

Comment:

UPDATE: The initial motivation for this ticket has been resolved.  The
 [https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test
 installation matrix] now contains text-mode versions of the
 [https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_Update_Bootloade...
 update bootloader] and
 [https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_Skip_Bootloader_...
 skip bootloader] tests.

I'm moving this ticket into the F-15 branch so we can continue to track
 updating the cases to better detect whether the bootloader was
 '''skipped''' or '''updated''' (see comment:8).

[fedora-badges]

#86: Add test coverage for text-mode upgrade
--------------------------+-------------------------------------------------
  Reporter:  jlaska       |       Owner:  rhe          
      Type:  enhancement  |      Status:  assigned     
  Priority:  major        |   Milestone:  Fedora 15    
 Component:  Wiki         |     Version:               
Resolution:               |    Keywords:  retrospective
--------------------------+-------------------------------------------------
Comment (by jlaska):

Replying to [comment:9 jlaska]:
> I'm moving this ticket into the F-15 branch so we can continue to track

updating the cases to better detect whether the bootloader was
 '''skipped''' or '''updated''' (see comment:8).

Just doing some ticket cleanup.  Hurry, are you okay with closing this
 ticket out ... and creating a new ticket (if needed) to resolve any
 remaining bootloader upgrade issues?

[fedora-badges]

#86: Add test coverage for text-mode upgrade
--------------------------+-------------------------------------------------
  Reporter:  jlaska       |       Owner:  rhe          
      Type:  enhancement  |      Status:  closed       
  Priority:  major        |   Milestone:  Fedora 15    
 Component:  Wiki         |     Version:               
Resolution:  fixed        |    Keywords:  retrospective
--------------------------+-------------------------------------------------
Changes (by rhe):

* status:  assigned => closed
  * resolution:  => fixed

Comment:

Replying to [comment:10 jlaska]:
> Replying to [comment:9 jlaska]:
> > I'm moving this ticket into the F-15 branch so we can continue to

track updating the cases to better detect whether the bootloader was
 '''skipped''' or '''updated''' (see comment:8).
> Just doing some ticket cleanup.  Hurry, are you okay with closing this

ticket out ... and creating a new ticket (if needed) to resolve any
 remaining bootloader upgrade issues?

Thanks, I'm closing this ticket and open another:
 [https://fedorahosted.org/fedora-qa/ticket/198 Ticket #198]