Proposed release criteria revisions

Fri Mar 18 13:15:01 UTC 2011

On Thu, 2011-03-17 at 13:16 -0700, Adam Williamson wrote:
> Can everyone let me know what they think of these changes? Thanks!
> 
> There may be a few more coming later as I go through the blocker meeting
> logs.

I hope you don't mind ... but my brain+eyes needed to see a patch
(available at http://fpaste.org/PX5u/).

> --- Fedora_15_Alpha_Release_Criteria.wiki	2011-03-18 08:36:31.976770141 -0400
> +++ Draft_Alpha_criteria_revision.wiki	2011-03-18 08:36:31.511803151 -0400
> @@ -8,6 +8,8 @@
>  == Alpha Release Requirements ==
>  In order to be the released to the general public, the Alpha Candidate (RC) must meet '''all''' of the following criteria.  This is intended to make the decision process as clear and straightforward as possible.  ''Mostly met'' items are ''incomplete'' until they are ''met.''  Optional and ''nice to have'' items should not be included in this list.
>  
> +There may be times where a requirement is unmet only in a particular configuration, such as with some keyboard layouts but not others, or if a particular character is used in a username, password or passphrase. In such cases, the release team should use their judgment and refer to precedent to determine whether or not the issue should be considered to block the release. They should consider the number of users likely to be affected by the issue, the severity of the case when the issue is encountered, and the ease or otherwise with which the issue can be avoided by both informed and uninformed users.

For me, this nicely captures the additional criteria we need to consider
when faced with sometimes fuzzy issues.  I tend to be a list person, so
I'd probably split the tail-end of this into a list, certainly not
required though.

>  # All bugs blocking the [https://bugzilla.redhat.com/showdependencytree.cgi?id=f15alpha&hide_resolved=1 Alpha tracker] must be [[BugZappers/BugStatusWorkFlow#CLOSED|CLOSED]]
>  # There must be no file conflicts (cases where the files in some packages conflict but the packages have explicit Conflicts: tags are acceptable) or unresolved package dependencies during a media-based (CD/DVD) install
>  # All dedicated installer images must boot to the graphical boot menu and allow the user to select install options. If no option is selected, the installer should load after a reasonable timeout
> @@ -21,7 +23,8 @@
>  # The installer must be able to complete an installation using the entire disk, existing free space, or existing Linux partitions methods, with or without encryption enabled
>  # The rescue mode of the installer must start successfully and be able to detect and mount an existing default installation
>  # The installer must be able to report failures to Bugzilla, with appropriate information included
> -# In most cases, the installed system must boot to a functional graphical environment without user intervention (see [[Blocker_Bug_FAQ#Hardware_and_local_configuration_dependent_issues|Blocker_Bug_FAQ]])
> +# In most cases (see [[Blocker_Bug_FAQ#Hardware_and_local_configuration_dependent_issues|Blocker_Bug_FAQ]]), a system installed according to any of the above criteria (or the appropriate Beta or Final criteria, when applying this criterion to those releases) must boot to the 'firstboot' utility on the first boot after installation, without unintended user intervention. This includes correctly accessing any encrypted partitions when the correct passphrase is supplied. The firstboot utility must be able to create a working user account
> +# Following on from the previous criterion, after firstboot is completed and on subsequent boots, a system installed according to any of the above criteria (or the appropriate Beta or Final criteria, when applying this criterion to those releases) must boot to a working graphical environment without unintended user intervention. This includes correctly accessing any encrypted partitions when the correct passphrase is supplied

What does "without unintended user intervention" mean?  Is that
distinguishing between manual installs, and unattended installations
where firstboot --disabled?

Why is "This includes correctly accessing any encrypted partitions when
the correct passphrase is supplied." included in both the firstboot and
the graphical login criteria?  Will it cover the same situations if that
statement is listed only in the second criteria addition (the must boot
to graphical criteria)?

>  # When booting a system installed without a graphical environment, or when using a correct configuration setting to cause an installed system to boot in non-graphical mode, the system should boot to a state where it is possible to log in through at least one of the default virtual consoles
>  # It must be possible to run the default web browser and a terminal application from the default desktop environment. The web browser must be able to download files, load extensions, and log into [https://admin.fedoraproject.org/accounts/ FAS]
>  # The installed system must be able to download and install updates with yum and PackageKit
> 
> --- Fedora_15_Beta_Release_Criteria.wiki	2011-03-18 08:36:33.307675652 -0400
> +++ Draft_Beta_criteria_revision.wiki	2011-03-18 08:36:32.797711857 -0400
> @@ -7,6 +7,8 @@
>  == Beta Release Requirements ==
>  In order to be the released to the general public, the Beta Candidate (RC) must meet '''all''' of the following criteria.  This is intended to make the decision process as clear and straightforward as possible.  ''Mostly met'' items are ''incomplete'' until they are ''met.''  Optional and ''nice to have'' items should not be included in this list.
>  
> +There may be times where a requirement is unmet only in a particular configuration, such as with some keyboard layouts but not others, or if a particular character is used in a username, password or passphrase. In such cases, the release team should use their judgment and refer to precedent to determine whether or not the issue should be considered to block the release. They should consider the number of users likely to be affected by the issue, the severity of the case when the issue is encountered, and the ease or otherwise with which the issue can be avoided by both informed and uninformed users. 
> +
>  Release Requirements:
>  # All [[Fedora 15 Alpha Release Criteria]] must be met
>  # All bugs blocking the [https://bugzilla.redhat.com/showdependencytree.cgi?id=f15beta&hide_resolved=1 Beta tracker] must be [[BugZappers/BugStatusWorkFlow#CLOSED|CLOSED]]
> @@ -18,16 +20,17 @@
>  # The installer must be able to use all kickstart delivery methods
>  # The installer must be able to create and install to software, hardware or BIOS RAID-0, RAID-1 or RAID-5 partitions for anything except /boot
>  # The installer must be able to successfully complete an upgrade installation from a clean, fully updated default installation of the previous stable Fedora release, either via preupgrade or by booting to the installer manually
> +# Any installation method or process designed to run unattended must do so (there should be no prompts requiring user intervention)

Thanks for taking a stab at this ... this has languished on my TODO list
for too long.  We've discussed this one several times, and I can't think
of any clever wording to avoid pulling in *every* obscure kickstart
scenario.

"Any installation method or process" ... we capture installation methods
elsewhere in the Alpha, Beta and Final criteria.  Howabout we leave
installation method out of these additions, and keep it focused on "Any
installation designed to run unattended..."?  Does that lose specific
scenarios you were wanting to include?

As worded, "designed to run unattended", includes any traceback
encountered during an unattended install, regardless of whether the
kickstart file is formatted correctly, or doing sane things?  How can we
distinguish between anaconda failures (captured by *many* other
criteria), vs unattended install failures.  Hopefully, I'm making sense
here. :)

This isn't great, but what about: "The installer must correctly honor
any properly formatted unattended installation configuration and, if
directed, install the system with no user intervention."

>  # The rescue mode of the installer must be able to detect and mount (read-write and read-only) LVM, encrypted, and RAID (BIOS, hardware, and software) installations
> -# When booting a system installed without a graphical environment, or when using a correct configuration setting to cause an installed system to boot in non-graphical mode, the system should provide a working login prompt without any user intervention (aside from the firstboot utility) when boot is complete, and all virtual consoles intended to provide a working login prompt should do so
> +# When booting a system installed without a graphical environment, or when using a correct configuration setting to cause an installed system to boot in non-graphical mode, the system should provide a working login prompt without any unintended user intervention when boot is complete, and all virtual consoles intended to provide a working login prompt should do so

Same clarification as stated earlier around the scenarios included by
the statement, "without any unintended user intervention".

>  # The release must boot successfully as a virtual guest in a situation where the virtual host is running the same release (using Fedora's current preferred virtualization technology)
>  # The release must boot successfully as a virtual guest in a situation where the virtual host is running the previous stable Fedora release (using Fedora's current preferred virtualization technology)
>  # The release must boot successfully as a virtual guest in a situation where the virtual host is running a supported Xen implementation
>  # In most cases, the installed system must be able to play back sound with gstreamer-based applications (see [[Blocker_Bug_FAQ#Hardware_and_local_configuration_dependent_issues|Blocker_Bug_FAQ]])
> -# No part of the default desktop's panel (or equivalent) configuration should crash on boot of the installed system using default installation choices
> +# No part of the default desktop's panel (or equivalent) configuration should crash or be entirely non-functional on boot of the installed system using default installation choices

Ack

>  # Automatic mounting on insertion of removable media must work in the default desktop environment
>  # The desktop default update manager must not periodically check for updates when the system is booted live, but must periodically check for updates when running on an installed system
> -# The desktop's offered mechanisms for shutting down, logging out and rebooting must work
> +# The desktop's offered mechanisms (if any) for shutting down, logging out and rebooting must work

Ack

>  <!-- 
>  IF we need this ... we fail
>  # The QA team has an installable testable Release Candidate for at least two full days (48 hours).
> 
> --- Fedora_15_Final_Release_Criteria.wiki	2011-03-18 08:36:34.296605443 -0400
> +++ Draft_Final_criteria_revision.wiki	2011-03-18 08:36:33.837638029 -0400
> @@ -5,6 +5,8 @@
>  == Final Release Requirements ==
>  In order to be the released to the general public, the Final Candidate (RC) must meet '''all''' of the following criteria.  This is intended to make the decision process as clear and straightforward as possible.  ''Mostly met'' items are ''incomplete'' until they are ''met.''  Optional and ''nice to have'' items should not be included in this list.
>  
> +There may be times where a requirement is unmet only in a particular configuration, such as with some keyboard layouts but not others, or if a particular character is used in a username, password or passphrase. In such cases, the release team should use their judgment and refer to precedent to determine whether or not the issue should be considered to block the release. They should consider the number of users likely to be affected by the issue, the severity of the case when the issue is encountered, and the ease or otherwise with which the issue can be avoided by both informed and uninformed users. 
> +
>  Release Requirements:
>  # All [[Fedora 15 Beta Release Criteria]] must be met
>  # All bugs blocking the [https://bugzilla.redhat.com/showdependencytree.cgi?id=f15blocker&hide_resolved=1 F15Blocker tracker] must be [[BugZappers/BugStatusWorkFlow#CLOSED|CLOSED]]
> @@ -18,13 +20,13 @@
>  # The installed system must run normally if the user chooses to install without SELinux
>  # Menu sanity - the following criteria refer to both a live image and default installed system
>  #* All ''Applications'' listed in the desktop menus must have icons which have a consistent appearance and sufficiently high resolution to avoid appearing blurry
> -#* All applications listed under the ''Applications'' menu must start successfully
> -#* All applications listed under the ''Applications'' menu must withstand a basic functionality test and not crash after a few minutes of normal use.  They must also have working ''Help'' and ''Help -> About'' menu items
> -#* There must be no ''Other'' menu
> +#* All applications listed under the ''Applications'' menu or category must start successfully
> +#* All applications listed under the ''Applications'' menu or category must withstand a basic functionality test and not crash after a few minutes of normal use.  They must also have working ''Help'' and ''Help -> About'' menu items
> +#* There must be no ''Other'' menu or category

Good catch, ack.

>  #* No application may unintentionally appear twice in the menus. In particular, items under ''System'' must not appear under ''Applications''
> -# All elements of the default panel configuration must be functional
> +# All elements of the default panel (or equivalent) configuration must function correctly in common use

I was confused this might be a DUP of the beta criteria at first.  But I
recall now that this captures *all* Desktop Environments, not just the
default.

>  # Saving passwords in the desktop default keyring (if the desktop implements one), and retrieving passwords from the keyring, must work
> -# The proposed final Fedora artwork is included and enabled by default for the installer, graphical boot, firstboot, graphical login and desktop background.  All Fedora artwork must be consistent with the proposed final theme, and if any artwork contains a graphical version number, the version number used must match the Fedora release number.  Generic release artwork (e.g. Alpha, Beta, Development) must not be used for the final release.
> +# The proposed final Fedora artwork must be included and enabled by default for the installer, graphical boot, firstboot, graphical login and desktop background.  All Fedora artwork must be consistent with the proposed final theme, and if any artwork contains a graphical version number, the version number used must match the Fedora release number.  Generic release artwork (e.g. Alpha, Beta, Development) must not be used for the final release

Ack

>  == Final Blocker Bugs ==
>  A bug is considered a [https://bugzilla.redhat.com/showdependencytree.cgi?id=f15blocker&hide_resolved=1 Final Blocker Bug] if '''any''' of the following criteria are met:

Thanks,
James
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/test/attachments/20110318/365eefa9/attachment.bin 