Final criteria/Test cases interconnection

Tue Dec 4 16:28:04 UTC 2012

Final release criteria:
http://fedoraproject.org/wiki/Fedora_18_Final_Release_Criteria

I went through all criteria and connect them to test cases. I found
following problems:

We don't have TCs for: 
* The release live images must properly support mounting and using a
persistent storage overlay for the entire system and/or one for
the /home partition, if such an overlay or overlays have been correctly
written to the medium from which the image is booted
* All critical path actions on release-blocking desktop environments
should correctly display all sufficiently complete translations
available for use - I don't think that this one is needed
* All known bugs that can cause corruption of user data must be fixed or
documented at Common F18 bugs - it doesn't make sence to have TC for
this one.
* No notices or alerts about pre-release status should be present -
there is no TC for anaconda
* The release must contain no known security bugs of 'important' or
higher impact according to the Red Hat severity classification scale
which cannot be satisfactorily resolved by a package update (e.g. issues
during installation)" - I don't think we need TC for this.
* A Package-x-generic-16.pngspin-kickstarts package which contains the
exact kickstart files used to build the release must be present in the
release repository. The included kickstarts must define the correct set
of release repositories
* The final branded release notes from the Documentation team must be
present on ISO media and the appropriately versioned generic release
notes must be available in the online release repository
* A Package-x-generic-16.pngfedora-release package containing the
correct names, information and repository configuration for a final
Fedora release (as opposed to a pre-release) must be present on ISO
media while the appropriately versioned
Package-x-generic-16.pnggeneric-release package must be available in the
online release repository

Obsolete TCs:
* Hardware and BIOS RAID TCs -  Do we support those now?
https://fedoraproject.org/wiki/QA:Testcase_dualboot_with_windows

Other problems:
* I'm not sure which network-attached storage devices are supported. We
have only this 
https://fedoraproject.org/wiki/QA:Testcase_EC2_AMI_Validation

Here is summary of all criteria/TC interconnections:
1. All Fedora 18 Beta Release Criteria must be met 
2. All bugs blocking the F18Blocker tracker must be CLOSED 
3. If there is an embedded checksum in the image, it must match. If
there is a related UI element displayed after booting the image, it must
work and display the correct result -
https://fedoraproject.org/wiki/QA:Testcase_Mediakit_ISO_Checksums
4. The release live images must properly support mounting and using a
persistent storage overlay for the entire system and/or one for
the /home partition, if such an overlay or overlays have been correctly
written to the medium from which the image is booted - we don't have
test case for this
5. The installer must be able to use all supported local and remote
package source options - We have TC for: DVD, mirrorlist, http, ftp, nfs
and nfsiso
6. The installer must be able to complete an installation using any
network-attached storage devices (e.g. iSCSI, FCoE, Fibre Channel) -
Only TC we have now is
https://fedoraproject.org/wiki/QA:Testcase_EC2_AMI_Validation Are other
devices supported in F18?
7. The installer must be able to complete an installation using all
supported interfaces - We have TCs for: graphical, basic video driver,
text, vnc and serial console
8. The installer must be able to create and install to any workable
partition layout using any file system offered in a default installer
configuration, LVM, software, hardware or BIOS RAID, or combination of
the above - We have TCs for: custom partitioning, software, hardware and
BIOS RAID. Hardware and BIOS RAID TCs are obsolete.
9. The installer must be able to install into free space alongside an
existing clean single-partition Windows installation and either install
a bootloader which can boot into the Windows installation, or leave the
Windows bootloader untouched and working -
https://fedoraproject.org/wiki/QA:Testcase_dualboot_with_windows it is
obsolete.
10. The installer must be able to use an installer update image
retrieved from removable media, remote installation source and HTTP
server - We have TCs for: URL, inst. source and local media
11. For each one of the release-blocking package sets ('minimal', and
the package sets for each one of the release-blocking desktops), it must
be possible to successfully complete an upgrade from a fully updated
installation of the previous stable Fedora release with that package set
installed, using all officially recommended upgrade mechanisms. The
upgraded system must meet all release criteria -  The only TC we have is
https://fedoraproject.org/wiki/QA:Testcase_upgrade_fedup_cli_previous_desktop
12. The release must boot successfully as Xen DomU with releases
providing a functional, supported Xen Dom0 and widely used cloud
providers utilizing Xen. This does not include any issues limited to the
release functioning as Xen Dom0 -
https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_Xen_Para_Virt
13. All services in a default install must start properly -
https://fedoraproject.org/wiki/QA:Testcase_Services_start
14. All critical path actions on release-blocking desktop environments
should correctly display all sufficiently complete translations
available for use - I don't think we have any TC for this.
15. In most cases, there must be no SELinux 'AVC: denied' messages or
abrt crash notifications on initial boot and subsequent login (see
Blocker_Bug_FAQ) -
https://fedoraproject.org/wiki/QA:Testcase_desktop_error_checks
16. All known bugs that can cause corruption of user data must be fixed
or documented at Common F18 bugs - it doesn't make sence to have TC for
this one.
17. Menu sanity - the following criteria refer to both a live image and
default installed system, and to all release-blocking desktops 
	* All Applications listed in the system menus (or equivalent) must have
icons which have a consistent appearance and sufficiently high
resolution to avoid appearing blurry 
	* All applications listed under the Applications menu or category must
start successfully 
	* All applications listed under the Applications menu or category must
withstand a basic functionality test and not crash after a few minutes
of normal use. They must also have working Help and Help -> About menu
items 
	* No application may unintentionally appear twice in the menus. In
particular, items under System must not appear under Applications -
There are 'menu' TCs in Desktop matrix
18. All elements of the default panel (or equivalent) configuration in
all release-blocking desktops must function correctly in common use -
there are two TCs (basic and advanced) in Desktop matrix
19. Saving passwords in the desktop default keyring (if the desktop
implements one), and retrieving passwords from the keyring, must work
for all release-blocking desktops -
https://fedoraproject.org/wiki/QA:Testcase_desktop_keyring
20. The proposed final Fedora artwork must be included and enabled by
default for all graphical bootloaders and the desktop background. All
Fedora artwork must be consistent with the proposed final theme - We
have two TCs in Base matrix - firstboot and startup (this one is for
backgrounds of *dm and desktop)
21. No notices or alerts about pre-release status should be present -
Same as 20. TC for anaconda is missing.
22. The release must contain no known security bugs of 'important' or
higher impact according to the Red Hat severity classification scale
which cannot be satisfactorily resolved by a package update (e.g. issues
during installation)" - I don't think we need TC for this.
23. A Package-x-generic-16.pngspin-kickstarts package which contains the
exact kickstart files used to build the release must be present in the
release repository. The included kickstarts must define the correct set
of release repositories -  We don't have a TC for this.
24. The final branded release notes from the Documentation team must be
present on ISO media and the appropriately versioned generic release
notes must be available in the online release repository - There is no
TC for this.
25. A Package-x-generic-16.pngfedora-release package containing the
correct names, information and repository configuration for a final
Fedora release (as opposed to a pre-release) must be present on ISO
media while the appropriately versioned
Package-x-generic-16.pnggeneric-release package must be available in the
online release repository - No TC