10:54 a.m.
This is another attempt at creating a new release criterion specific for
graphical package managers. We already discussed it in November, but we
couldn't agree easily on some particular details, and then I was gone for a
long time:
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.or...
Here is my new attempt. The new criterion text is longer, but I believe it
is overall somewhat weaker (to accommodate some of the previous concerns),
just more explicit. As before, please note that we already have package
management partly covered in the Basic criteria [1] and Beta criteria [2].
Please read those first, including footnotes. The following new criterion
is proposed against the Final milestone:
~~~~~~~~~~~~~~~~~~~~
The default graphical package manager for a given software type must
appropriately:
* Install, remove and update software
* List locally-installed software coming from the official Fedora
repositories
* List available software (possibly in categories, a curated list, etc)
* Display metadata relevant to the selected software (e.g. the description,
screenshots, the size)
* Start the selected installed software
* Configure software sources by enabling/disabling pre-defined official
repositories and then adjust the available software pool accordingly
* Notify the user when system updates are available (taking into account
the intended frequency of such notifications)
The following must hold true:
* When multiple operations (covered by this criterion) are requested, all
of them must finish correctly. (It's also valid to inform the user that a
certain operation is not available at that moment, see the "Supported
functionality and design decisions" note).
* The displayed state of software or software sources must not differ from
their actual state. (E.g. an RPM package must not be shown as installed
when it is not, a repository must not be shown as disabled or missing when
it is enabled, etc).
* Usual GUI interactivity must not break operations covered by this
criterion. (E.g. when the GUI allows the user to click some buttons while
an operation is running, it must not break that operation).
* The package manager must never make the system enter an inconsistent or
unbootable state. (E.g. damage the local software database, remove wrong
system files, break the bootloader, etc).
Note: Supported functionality and design decisions
All requirements apply only if such a functionality is supported by the
package manager; it does not imply that the package manager must support
all this functionality. The requirements don't apply if some functionality
is intentionally modified as a design decision (e.g. if some software
sources can't be disabled as an intentional precaution to users breaking
their system). If there is a bug violating the design decisions in one of
the covered areas (e.g. a software source which is supposed to be always
enabled can be disabled), it's up to the blocker review team to consider
whether this bug makes the user experience or system safety considerably
worse.
Note: Refer to Basic footnotes
The footnotes for the [similar Basic criterion covering console tools][1]
regarding software types, 'appropriate' behavior, scope and so on are all
generally applicable to this criterion also.
~~~~~~~~~~~~~~~~~~~~
If you compare it to the previous proposal, the new one avoids talking
about "sequential or concurrent" operations. That leaves some more wiggle
room when arguing that specific approaches are too niche (Frantisek's
concern), i.e. it no longer clearly covers these approaches completely.
However, it still makes sure that multiple operations are covered at least
at a basic level (which should cover our famous "install and then remove
the same package"). The tool can also say "no can do", and when that
happens to be some internal error message, we can discuss how clear it is
to the user.
I updated the "list installed software" and "configure sources"
requirements with suggestions from Ben.
It also explicitly mentions that just basic clicking through GUI must not
abort running operations (which was something that got good consensus). And
adds an obvious statement about not breaking the system, which was implied
in the past but I assumed it's better to have it clearly visible (this one
could be moved to Beta, possibly).
Finally it adds a requirement that the package manager must not mislead the
user by showing e.g. something as installed when it is not. You can say
this was also implied in the past (otherwise the 'install' or 'list'
operation is not working correctly). but again, this is just to avoid "it
works well, just displays it wrongly" arguments in the future.
What do you think?
[1]
https://fedoraproject.org/wiki/Basic_Release_Criteria#software-install-re...
[2]
https://fedoraproject.org/wiki/Fedora_35_Beta_Release_Criteria#Installing...
12:48 p.m.
On Thu, Jan 20, 2022 at 9:55 AM Kamil Paral <kparal(a)redhat.com> wrote:
This is another attempt at creating a new release criterion specific for graphical
package managers. We already discussed it in November, but we couldn't agree easily on
some particular details, and then I was gone for a long time:
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.or...
Here is my new attempt. The new criterion text is longer, but I believe it is overall
somewhat weaker (to accommodate some of the previous concerns), just more explicit. As
before, please note that we already have package management partly covered in the Basic
criteria [1] and Beta criteria [2]. Please read those first, including footnotes. The
following new criterion is proposed against the Final milestone:
~~~~~~~~~~~~~~~~~~~~
The default graphical package manager for a given software type must appropriately:
* Install, remove and update software
* List locally-installed software coming from the official Fedora repositories
* List available software (possibly in categories, a curated list, etc)
* Display metadata relevant to the selected software (e.g. the description, screenshots,
the size)
* Start the selected installed software
* Configure software sources by enabling/disabling pre-defined official repositories and
then adjust the available software pool accordingly
* Notify the user when system updates are available (taking into account the intended
frequency of such notifications)
The following must hold true:
* When multiple operations (covered by this criterion) are requested, all of them must
finish correctly. (It's also valid to inform the user that a certain operation is not
available at that moment, see the "Supported functionality and design decisions"
note).
* The displayed state of software or software sources must not differ from their actual
state. (E.g. an RPM package must not be shown as installed when it is not, a repository
must not be shown as disabled or missing when it is enabled, etc).
* Usual GUI interactivity must not break operations covered by this criterion. (E.g. when
the GUI allows the user to click some buttons while an operation is running, it must not
break that operation).
* The package manager must never make the system enter an inconsistent or unbootable
state. (E.g. damage the local software database, remove wrong system files, break the
bootloader, etc).
Note: Supported functionality and design decisions
All requirements apply only if such a functionality is supported by the package manager;
it does not imply that the package manager must support all this functionality. The
requirements don't apply if some functionality is intentionally modified as a design
decision (e.g. if some software sources can't be disabled as an intentional precaution
to users breaking their system). If there is a bug violating the design decisions in one
of the covered areas (e.g. a software source which is supposed to be always enabled can be
disabled), it's up to the blocker review team to consider whether this bug makes the
user experience or system safety considerably worse.
Note: Refer to Basic footnotes
The footnotes for the [similar Basic criterion covering console tools][1] regarding
software types, 'appropriate' behavior, scope and so on are all generally
applicable to this criterion also.
~~~~~~~~~~~~~~~~~~~~
If you compare it to the previous proposal, the new one avoids talking about
"sequential or concurrent" operations. That leaves some more wiggle room when
arguing that specific approaches are too niche (Frantisek's concern), i.e. it no
longer clearly covers these approaches completely. However, it still makes sure that
multiple operations are covered at least at a basic level (which should cover our famous
"install and then remove the same package"). The tool can also say "no can
do", and when that happens to be some internal error message, we can discuss how
clear it is to the user.
I updated the "list installed software" and "configure sources"
requirements with suggestions from Ben.
It also explicitly mentions that just basic clicking through GUI must not abort running
operations (which was something that got good consensus). And adds an obvious statement
about not breaking the system, which was implied in the past but I assumed it's better
to have it clearly visible (this one could be moved to Beta, possibly).
Finally it adds a requirement that the package manager must not mislead the user by
showing e.g. something as installed when it is not. You can say this was also implied in
the past (otherwise the 'install' or 'list' operation is not working
correctly). but again, this is just to avoid "it works well, just displays it
wrongly" arguments in the future.
What do you think?
Looks good to me.
--
Chris Murphy
1:17 p.m.
On 2022-01-20 11:54 a.m., Kamil Paral wrote:
This is another attempt at creating a new release criterion specific
for graphical package managers. We already discussed it in November,
but we couldn't agree easily on some particular details, and then I
was gone for a long time:
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.or...
Here is my new attempt. The new criterion text is longer, but I
believe it is overall somewhat weaker (to accommodate some of the
previous concerns), just more explicit. As before, please note that we
already have package management partly covered in the Basic criteria
[1] and Beta criteria [2]. Please read those first, including
footnotes. The following new criterion is proposed against the Final
milestone:
~~~~~~~~~~~~~~~~~~~~
The default graphical package manager for a given software type must
appropriately:
* Install, remove and update software
* List locally-installed software coming from the official Fedora
repositories
* List available software (possibly in categories, a curated list, etc)
* Display metadata relevant to the selected software (e.g. the
description, screenshots, the size)
* Start the selected installed software
* Configure software sources by enabling/disabling pre-defined
official repositories and then adjust the available software pool
accordingly
* Notify the user when system updates are available (taking into
account the intended frequency of such notifications)
The following must hold true:
* When multiple operations (covered by this criterion) are requested,
all of them must finish correctly. (It's also valid to inform the user
that a certain operation is not available at that moment, see the
"Supported functionality and design decisions" note).
* The displayed state of software or software sources must not differ
from their actual state. (E.g. an RPM package must not be shown as
installed when it is not, a repository must not be shown as disabled
or missing when it is enabled, etc).
* Usual GUI interactivity must not break operations covered by this
criterion. (E.g. when the GUI allows the user to click some buttons
while an operation is running, it must not break that operation).
* The package manager must never make the system enter an inconsistent
or unbootable state. (E.g. damage the local software database, remove
wrong system files, break the bootloader, etc).
Note: Supported functionality and design decisions
All requirements apply only if such a functionality is supported by
the package manager; it does not imply that the package manager must
support all this functionality. The requirements don't apply if some
functionality is intentionally modified as a design decision (e.g. if
some software sources can't be disabled as an intentional precaution
to users breaking their system). If there is a bug violating the
design decisions in one of the covered areas (e.g. a software source
which is supposed to be always enabled can be disabled), it's up to
the blocker review team to consider whether this bug makes the user
experience or system safety considerably worse.
Note: Refer to Basic footnotes
The footnotes for the [similar Basic criterion covering console
tools][1] regarding software types, 'appropriate' behavior, scope and
so on are all generally applicable to this criterion also.
~~~~~~~~~~~~~~~~~~~~
If you compare it to the previous proposal, the new one avoids talking
about "sequential or concurrent" operations. That leaves some more
wiggle room when arguing that specific approaches are too niche
(Frantisek's concern), i.e. it no longer clearly covers these
approaches completely. However, it still makes sure that multiple
operations are covered at least at a basic level (which should cover
our famous "install and then remove the same package"). The tool can
also say "no can do", and when that happens to be some internal error
message, we can discuss how clear it is to the user.
I updated the "list installed software" and "configure sources"
requirements with suggestions from Ben.
It also explicitly mentions that just basic clicking through GUI must
not abort running operations (which was something that got good
consensus). And adds an obvious statement about not breaking the
system, which was implied in the past but I assumed it's better to
have it clearly visible (this one could be moved to Beta, possibly).
Finally it adds a requirement that the package manager must not
mislead the user by showing e.g. something as installed when it is
not. You can say this was also implied in the past (otherwise the
'install' or 'list' operation is not working correctly). but again,
this is just to avoid "it works well, just displays it wrongly"
arguments in the future.
What do you think?
[1]
https://fedoraproject.org/wiki/Basic_Release_Criteria#software-install-re...
[2]
https://fedoraproject.org/wiki/Fedora_35_Beta_Release_Criteria#Installing...
I've done generic, app-specific and custom installers and lots of
install environments on a lot of architectures for about 45 years of my
50 years working as a mission-critical developer and maintainer. There
a lots of people who do not understand the Unix way of doing things or
how to take advantage of the Unix filesystem to do things a lot better
than Windows, and try to make installation or upgrades behave more like
ancient Windows patches. There are also a growing number of Fedora
users who are now discarding the GUI updater in favour of dnf installs,
primarily due to the GUI updater having extreme and unnecessary reboot
requirements.
The suggested criteria list is pretty generic, and offhand I can't think
of a current GUI or CLI installer that does not conform to these
requirements.
What this list does NOT cover are the pain points of the current
product, such as avoiding reboots. Currently, the Fedora GUI installer
always reboots twice, even though 80% of the updates do not require
reboots at all, and similar to the Debian-based distros, tools such as
the needs-restarting component of the yum-utils package exist to show
that shared libs or open files are out-of-date and either app or full
system reboots are required to put the system into a supportable state.
Unlike other distros, there are certain apps, like parts of the Gnome
desktop and Firefox, that do not respect a policy of no API changes in a
given release, and these apps tend to break routinely because of this
broken development model being considered acceptable. If I look at
Ubuntu practices for example, they introduce a needs-rebooting flag that
is set when a memory-mapped file is altered, or when an API change is
intentionally introduced. However, having multiple versions of shared
libs in use at the same time is not an error, but a really good
Unix/Linux feature that is not a valid criteria for a reboot. The
current GUI installer behaves more like Windows, and does not allow this
situation when it most certainly should. Fedora should consider fixing
these packages that routinely allow developer errors instead of
downloading the resulting problems to the end user, and the installer
tools should not be aiding and abetting bad development practices.
Just my humble opinion as a past master of installers or all types and
sizes.
--
John Mellor
1:55 p.m.
On Thu, Jan 20, 2022 at 12:18 PM John Mellor <jmellor(a)rogers.com> wrote:
I've done generic, app-specific and custom installers and lots
of
install environments on a lot of architectures for about 45 years of my
50 years working as a mission-critical developer and maintainer. There
a lots of people who do not understand the Unix way of doing things or
how to take advantage of the Unix filesystem to do things a lot better
than Windows, and try to make installation or upgrades behave more like
ancient Windows patches. There are also a growing number of Fedora
users who are now discarding the GUI updater in favour of dnf installs,
primarily due to the GUI updater having extreme and unnecessary reboot
requirements.
Maybe, but it's probably off topic for the test@ list. I think instead
it belongs on the desktop@ list where it can be discussed by the
Workstation working group. The working group is due to revise the
product requirements document this month.
The central problem with reboots is, we don't have a sophisticated
enough process to know for sure whether the environment is in a known
good state. What if the kernel has had an oops or is tainted in a way
that could negatively impact the update? The current update strategy
is fragile - it's making modifications to the currently running
system, and if it were to fail it can leave the system broken in a
variety of really inconvenient ways. And frankly, no one wants to take
a chance on any of that, hence the first reboot is to ensure clean
kernel state, and a limited environment with reduced number of
services for performing the update with minimal chance of conflict.
No one has argued reboots are inherently mandatory, just that given
the resources, frankly it's cheaper to require the two reboots. There
are other ways of updating being explored to reduce the reboot to one.
And reduce update frequency so that the compulsory reboots aren't
happening more than once every couple weeks, hopefully.
--
Chris Murphy
5:21 p.m.
On Thu, 2022-01-20 at 14:17 -0500, John Mellor wrote:
The suggested criteria list is pretty generic, and offhand I
can't think
of a current GUI or CLI installer that does not conform to these
requirements.
Just before Fedora 35 was released, the KDE package manager did not
conform to several of them:
https://bugzilla.redhat.com/show_bug.cgi?id=2015809
https://bugzilla.redhat.com/show_bug.cgi?id=2011322
https://bugzilla.redhat.com/show_bug.cgi?id=2015491
https://bugzilla.redhat.com/show_bug.cgi?id=2011291
https://bugzilla.redhat.com/show_bug.cgi?id=2011333
It was largely these and a few other bugs that motivated us to draft
these criteria.
As I mentioned in my other mail, the point of the release criteria is
to specify a subset of expected functionality which is so important we
should block the release if it's not met. It's not *meant* to contain
anything novel or unexpected. *Everything* in the release criteria
should be "the way it's supposed to work already". What we're doing
with the release criteria is saying "if these things *don't* work the
way they're intended to, that's bad enough that we shouldn't ship the
release".
What this list does NOT cover are the pain points of the current
product, such as avoiding reboots.
Yes, because reboots are a part of the way it's supposed to work. It
would be wrong for the release criteria to say "there shouldn't be any
reboots" if some of the package managers the distribution has chosen to
use have chosen to require reboots on updates, and the distribution has
decided that it's fine with that (as we have). Release criteria are not
tools for advocacy about how we we like the software to work. That
should be done elsewhere.
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
2:10 p.m.
Great work, Kamil. I think this addresses most of the concerns. And
while I still don't love the idea of including concurrent actions, the
argument is good enough that I'll live with it. I have some pedantic
arguments about one of the things that must hold true, but that's
something to discuss socially. I'm +1 to the modified proposal.
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
1:20 p.m.
On 2022-01-20 11:54 a.m., Kamil Paral wrote:
This is another attempt at creating a new release criterion specific
for graphical package managers. We already discussed it in November,
but we couldn't agree easily on some particular details, and then I
was gone for a long time:
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.or...
Here is my new attempt. The new criterion text is longer, but I
believe it is overall somewhat weaker (to accommodate some of the
previous concerns), just more explicit. As before, please note that we
already have package management partly covered in the Basic criteria
[1] and Beta criteria [2]. Please read those first, including
footnotes. The following new criterion is proposed against the Final
milestone:
~~~~~~~~~~~~~~~~~~~~
The default graphical package manager for a given software type must
appropriately:
* Install, remove and update software
* List locally-installed software coming from the official Fedora
repositories
* List available software (possibly in categories, a curated list, etc)
* Display metadata relevant to the selected software (e.g. the
description, screenshots, the size)
* Start the selected installed software
* Configure software sources by enabling/disabling pre-defined
official repositories and then adjust the available software pool
accordingly
* Notify the user when system updates are available (taking into
account the intended frequency of such notifications)
The following must hold true:
* When multiple operations (covered by this criterion) are requested,
all of them must finish correctly. (It's also valid to inform the user
that a certain operation is not available at that moment, see the
"Supported functionality and design decisions" note).
* The displayed state of software or software sources must not differ
from their actual state. (E.g. an RPM package must not be shown as
installed when it is not, a repository must not be shown as disabled
or missing when it is enabled, etc).
* Usual GUI interactivity must not break operations covered by this
criterion. (E.g. when the GUI allows the user to click some buttons
while an operation is running, it must not break that operation).
* The package manager must never make the system enter an inconsistent
or unbootable state. (E.g. damage the local software database, remove
wrong system files, break the bootloader, etc).
Note: Supported functionality and design decisions
All requirements apply only if such a functionality is supported by
the package manager; it does not imply that the package manager must
support all this functionality. The requirements don't apply if some
functionality is intentionally modified as a design decision (e.g. if
some software sources can't be disabled as an intentional precaution
to users breaking their system). If there is a bug violating the
design decisions in one of the covered areas (e.g. a software source
which is supposed to be always enabled can be disabled), it's up to
the blocker review team to consider whether this bug makes the user
experience or system safety considerably worse.
Note: Refer to Basic footnotes
The footnotes for the [similar Basic criterion covering console
tools][1] regarding software types, 'appropriate' behavior, scope and
so on are all generally applicable to this criterion also.
~~~~~~~~~~~~~~~~~~~~
If you compare it to the previous proposal, the new one avoids talking
about "sequential or concurrent" operations. That leaves some more
wiggle room when arguing that specific approaches are too niche
(Frantisek's concern), i.e. it no longer clearly covers these
approaches completely. However, it still makes sure that multiple
operations are covered at least at a basic level (which should cover
our famous "install and then remove the same package"). The tool can
also say "no can do", and when that happens to be some internal error
message, we can discuss how clear it is to the user.
I updated the "list installed software" and "configure sources"
requirements with suggestions from Ben.
It also explicitly mentions that just basic clicking through GUI must
not abort running operations (which was something that got good
consensus). And adds an obvious statement about not breaking the
system, which was implied in the past but I assumed it's better to
have it clearly visible (this one could be moved to Beta, possibly).
Finally it adds a requirement that the package manager must not
mislead the user by showing e.g. something as installed when it is
not. You can say this was also implied in the past (otherwise the
'install' or 'list' operation is not working correctly). but again,
this is just to avoid "it works well, just displays it wrongly"
arguments in the future.
What do you think?
[1]
https://fedoraproject.org/wiki/Basic_Release_Criteria#software-install-re...
[2]
https://fedoraproject.org/wiki/Fedora_35_Beta_Release_Criteria#Installing...
I am unclear from this proposal on what is supposed to happen if a
package is not completely installed. E.g: a powerdown or other outage
causes the post-install script in an installed rpm to not be run. A few
packages like the kernel can take ten minutes to install properly, and
you may not have enough battery left to get that done when the power
goes out. The package and its dependencies are installed, but not
initialized correctly. Does it force an installer continuation or redo
at boot time for these partial installs, does in uninstall the app and
its whole chain of dependencies, or does it just leave things in a
corrupted state?
Similarly, if an already-installed package is corrupted and then
upgraded, what is the expected handling?
--
John ellor
4:54 p.m.
On Fri, 2022-01-21 at 14:20 -0500, John Mellor wrote:
>
I am unclear from this proposal on what is supposed to happen if a
package is not completely installed. E.g: a powerdown or other outage
causes the post-install script in an installed rpm to not be run. A few
packages like the kernel can take ten minutes to install properly, and
you may not have enough battery left to get that done when the power
goes out. The package and its dependencies are installed, but not
initialized correctly. Does it force an installer continuation or redo
at boot time for these partial installs, does in uninstall the app and
its whole chain of dependencies, or does it just leave things in a
corrupted state?
The release criteria aren't design documents. We're not defining the
intended behavior, exactly; we're defining a certain subset of known
expected behavior which we require to *work*.
The situations you describe are, essentially, "disaster recovery"
scenarios. What our current package managers do in this scenario is
"leave things in a corrupted state". That's been the case for years and
will likely continue to be so. It seems kinda pointless to have a
release criterion asserting that this is the case, and we do not do
'enforced-product-design-by-release-criteria' (where we make up release
criteria that describe how we'd *like* things to behave, and then try
and use that as a stick to make the developers make things actually
behave that way).
So, I'd say there isn't really anything useful we could do by writing
criteria around the scenario of interrupted updates.
It's worth noting that we do have one rpm-ostree based release-blocking
edition at present - IoT - and we may have more (e.g. Silverblue and/or
CoreOS) in future. For rpm-ostree based installs, we can and do assert
in the release criteria that rollbacks and rebases work, for recovery
from broken update attempts:
https://fedoraproject.org/wiki/Basic_Release_Criteria#rpm-ostree_requirem...
because that is how things are designed to work, and it's clearly
important enough that we should make it part of the release criteria
that things actually work that way. But for traditional RPM-based
installs, it doesn't really seem like something we can usefully address
in the criteria to me.
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
5:20 p.m.
On 2022-01-21 5:54 p.m., Adam Williamson wrote:
On Fri, 2022-01-21 at 14:20 -0500, John Mellor wrote:
> I am unclear from this proposal on what is supposed to happen if a
> package is not completely installed. E.g: a powerdown or other outage
> causes the post-install script in an installed rpm to not be run. A few
> packages like the kernel can take ten minutes to install properly, and
> you may not have enough battery left to get that done when the power
> goes out. The package and its dependencies are installed, but not
> initialized correctly. Does it force an installer continuation or redo
> at boot time for these partial installs, does in uninstall the app and
> its whole chain of dependencies, or does it just leave things in a
> corrupted state?
The release criteria aren't design documents. We're not defining the
intended behavior, exactly; we're defining a certain subset of known
expected behavior which we require to *work*.
The situations you describe are, essentially, "disaster recovery"
scenarios. What our current package managers do in this scenario is
"leave things in a corrupted state". That's been the case for years and
will likely continue to be so. It seems kinda pointless to have a
release criterion asserting that this is the case, and we do not do
'enforced-product-design-by-release-criteria' (where we make up release
criteria that describe how we'd *like* things to behave, and then try
and use that as a stick to make the developers make things actually
behave that way).
So, I'd say there isn't really anything useful we could do by writing
criteria around the scenario of interrupted updates.
It's worth noting that we do have one rpm-ostree based release-blocking
edition at present - IoT - and we may have more (e.g. Silverblue and/or
CoreOS) in future. For rpm-ostree based installs, we can and do assert
in the release criteria that rollbacks and rebases work, for recovery
from broken update attempts:
https://fedoraproject.org/wiki/Basic_Release_Criteria#rpm-ostree_requirem...
because that is how things are designed to work, and it's clearly
important enough that we should make it part of the release criteria
that things actually work that way. But for traditional RPM-based
installs, it doesn't really seem like something we can usefully address
in the criteria to me.
Ok, so would that allowance not violate 2 of the proposed criteria:
1. * The displayed state of software or software sources must not
differ from their actual state. (E.g. an RPM package must not be
shown as installed when it is not, a repository must not be shown as
disabled or missing when it is enabled, etc).
2. * The package manager must never make the system enter an
inconsistent or unbootable state. (E.g. damage the local software
database, remove wrong system files, break the bootloader, etc).
Or is the purpose of this document just to provide validation criteria
for the existing installer behaviour?
--
John Mellor
5:34 p.m.
On Fri, 2022-01-21 at 18:20 -0500, John Mellor wrote:
Ok, so would that allowance not violate 2 of the proposed criteria:
1. * The displayed state of software or software sources must not
differ from their actual state. (E.g. an RPM package must not be
shown as installed when it is not, a repository must not be shown as
disabled or missing when it is enabled, etc).
2. * The package manager must never make the system enter an
inconsistent or unbootable state. (E.g. damage the local software
database, remove wrong system files, break the bootloader, etc).
We probably should carve out an allowance there, in fact, yeah. Kamil,
what do you think?
Or is the purpose of this document just to provide validation criteria
for the existing installer behaviour?
Yeah, as I wrote, that's the point of release criteria: they define the
most important parts of existing behaviour. We're not designing *new*
desired behaviour here.
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
8:28 a.m.
On Sat, Jan 22, 2022 at 12:36 AM Adam Williamson <adamwill(a)fedoraproject.org>
wrote:
On Fri, 2022-01-21 at 18:20 -0500, John Mellor wrote:
>
> Ok, so would that allowance not violate 2 of the proposed criteria:
>
> 1. * The displayed state of software or software sources must not
> differ from their actual state. (E.g. an RPM package must not be
> shown as installed when it is not, a repository must not be shown as
> disabled or missing when it is enabled, etc).
> 2. * The package manager must never make the system enter an
> inconsistent or unbootable state. (E.g. damage the local software
> database, remove wrong system files, break the bootloader, etc).
We probably should carve out an allowance there, in fact, yeah. Kamil,
what do you think?
I think that I don't understand what the issue is :-) Can you re-phrase
"carve out an allowance" and be more specific?
If this is about power outages, I don't see any problem here. A power
outage is not the package manager's failure, and so those criteria don't
apply.
If this is about working with an already broken system (broken rpmdb or
similar), the package manager again didn't cause this state (it didn't
"make the system enter an inconsistent state"), and so again that criterion
doesn't apply. It could possibly display incorrect state of e.g. rpm
packages, if rpmdb is broken (but who can say what the actual state is,
when the db is broken?), but the same issue could then apply e.g. to "must
install, remove and update software".
We could add a footer "these criteria only apply when the system is in a
consistent state", but in that case we might need to add such a footer to
most of our criteria :-) E.g. a file browser must be able to copy files
only when the filesystem is in a consistent state.
I would expect that these cases would be considered a conditional blocker,
the condition being that the system was broken before, and therefore
waived, because on a broken system we can't guarantee anything.
Or was this about something different?
11:19 a.m.
On Mon, 2022-01-24 at 15:28 +0100, Kamil Paral wrote:
On Sat, Jan 22, 2022 at 12:36 AM Adam Williamson
<adamwill(a)fedoraproject.org>
wrote:
> On Fri, 2022-01-21 at 18:20 -0500, John Mellor wrote:
> >
> > Ok, so would that allowance not violate 2 of the proposed criteria:
> >
> > 1. * The displayed state of software or software sources must not
> > differ from their actual state. (E.g. an RPM package must not be
> > shown as installed when it is not, a repository must not be shown as
> > disabled or missing when it is enabled, etc).
> > 2. * The package manager must never make the system enter an
> > inconsistent or unbootable state. (E.g. damage the local software
> > database, remove wrong system files, break the bootloader, etc).
>
> We probably should carve out an allowance there, in fact, yeah. Kamil,
> what do you think?
>
I think that I don't understand what the issue is :-) Can you re-phrase
"carve out an allowance" and be more specific?
If this is about power outages, I don't see any problem here. A power
outage is not the package manager's failure, and so those criteria don't
apply.
I can see someone proposing it, though. "I lost power in the middle of
an update and now the system is unbootable". Whether it was "caused" by
the power failure or the package manager is kind of a matter of
perspective. I just figured it might be worth a footnote to explicitly
specify that such situations aren't covered, for clarity. It's
technically true that a system whose power got cut could fail all sorts
of criteria, I guess...it just feels more likely that someone might
think this criterion means the package manager is meant to handle it, I
guess.
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
4:02 a.m.
On Mon, Jan 24, 2022 at 6:19 PM Adam Williamson <adamwill(a)fedoraproject.org>
wrote:
On Mon, 2022-01-24 at 15:28 +0100, Kamil Paral wrote:
> If this is about power outages, I don't see any problem here. A power
> outage is not the package manager's failure, and so those criteria don't
> apply.
I can see someone proposing it, though. "I lost power in the middle of
an update and now the system is unbootable". Whether it was "caused" by
the power failure or the package manager is kind of a matter of
perspective. I just figured it might be worth a footnote to explicitly
specify that such situations aren't covered, for clarity. It's
technically true that a system whose power got cut could fail all sorts
of criteria, I guess...it just feels more likely that someone might
think this criterion means the package manager is meant to handle it, I
guess.
OK, so what about adding a note like this:
Note: Systems in an inconsistent state
While the package manager must not be the primary cause for breaking a
system (unbootable, invalid internal structures, etc), it doesn't have to
'''prevent''' these events from happening. So if there's e.g.
a power
outage during its operation or a package with harmful scriptlets is
installed, which breaks the system, this is not the fault of the package
manager and the criteria above are not considered violated. Similarly, when
the package manager operates on an already broken system (e.g. with an
inconsistent rpm database), the correct behavior cannot be guaranteed, and
therefore the criteria also don't apply.
7:58 a.m.
On 2022-01-25 05:02, Kamil Paral wrote:
On Mon, Jan 24, 2022 at 6:19 PM Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
On Mon, 2022-01-24 at 15:28 +0100, Kamil Paral wrote:
> If this is about power outages, I don't see any problem here. A
power
> outage is not the package manager's failure, and so those
criteria don't
> apply.
I can see someone proposing it, though. "I lost power in the middle of
an update and now the system is unbootable". Whether it was
"caused" by
the power failure or the package manager is kind of a matter of
perspective. I just figured it might be worth a footnote to explicitly
specify that such situations aren't covered, for clarity. It's
technically true that a system whose power got cut could fail all
sorts
of criteria, I guess...it just feels more likely that someone might
think this criterion means the package manager is meant to handle
it, I
guess.
OK, so what about adding a note like this:
Note: Systems in an inconsistent state
While the package manager must not be the primary cause for breaking a
system (unbootable, invalid internal structures, etc), it doesn't have
to '''prevent''' these events from happening. So if there's
e.g. a
power outage during its operation or a package with harmful scriptlets
is installed, which breaks the system, this is not the fault of the
package manager and the criteria above are not considered violated.
Similarly, when the package manager operates on an already broken
system (e.g. with an inconsistent rpm database), the correct behavior
cannot be guaranteed, and therefore the criteria also don't apply.
I'm ok with that or something similar, but it does point out the need to
fill a large gap in keeping the machine sane when something unexpected
happens. Perhaps F36 can expedite btrfs gui and cli tools to roll back
to the last known sane state in both the normal and diagnostic images.
You know, like the Sun time-slider on the gui and maybe even the Apple
time machine functionality.
--
John Mellor
9:53 a.m.
On Tue, Jan 25, 2022 at 3:00 PM John Mellor <jmellor(a)rogers.com> wrote:
I'm ok with that or something similar, but it does point out the
need to
fill a large gap in keeping the machine sane when something unexpected
happens. Perhaps F36 can expedite btrfs gui and cli tools to roll back to
the last known sane state in both the normal and diagnostic images.
You have to talk to the developers about that, not QA ;-)
3:10 a.m.
The criteria looks good to me, but I agree that it might be a double edged
sword to say that *The package manager must never make the system enter an
inconsistent or unbootable state* as suggested. An explanatory note is also
a good thing to have. However I am not convinced that the wording needs to
be as complex as proposed.
What if we said something like:
The package manager must never be the primary cause to damage a previously
sane installed system in any way.
On Tue, Jan 25, 2022 at 4:54 PM Kamil Paral <kparal(a)redhat.com> wrote:
On Tue, Jan 25, 2022 at 3:00 PM John Mellor
<jmellor(a)rogers.com> wrote:
> I'm ok with that or something similar, but it does point out the need to
> fill a large gap in keeping the machine sane when something unexpected
> happens. Perhaps F36 can expedite btrfs gui and cli tools to roll back to
> the last known sane state in both the normal and diagnostic images.
>
You have to talk to the developers about that, not QA ;-)
_______________________________________________
test mailing list -- test(a)lists.fedoraproject.org
To unsubscribe send an email to test-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
--
Lukáš Růžička
FEDORA QE, RHCE
Red Hat
<https://www.redhat.com>
Purkyňova 115
612 45 Brno - Královo Pole
lruzicka(a)redhat.com
TRIED AND PERSONALLY TESTED, ERGO TRUSTED. <https://redhat.com/trusted>
2 p.m.
On Tue, 2022-01-25 at 11:02 +0100, Kamil Paral wrote:
OK, so what about adding a note like this:
Note: Systems in an inconsistent state
While the package manager must not be the primary cause for breaking a
system (unbootable, invalid internal structures, etc), it doesn't have to
'''prevent''' these events from happening. So if there's e.g.
a power
outage during its operation or a package with harmful scriptlets is
installed, which breaks the system, this is not the fault of the package
manager and the criteria above are not considered violated. Similarly, when
the package manager operates on an already broken system (e.g. with an
inconsistent rpm database), the correct behavior cannot be guaranteed, and
therefore the criteria also don't apply.
That sounds fine to me. Lukas' is shorter, but yours to me is clearer
about the purpose of the note, so I like it more. Thanks!
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
2:48 a.m.
On Mon, Feb 7, 2022 at 9:05 PM Adam Williamson <adamwill(a)fedoraproject.org>
wrote:
That sounds fine to me. Lukas' is shorter, but yours to me is
clearer
about the purpose of the note, so I like it more. Thanks!
OK. I sent out the complete criterion proposal to Workstation and KDE lists:
https://lists.fedoraproject.org/archives/list/desktop@lists.fedoraproject...
https://lists.fedoraproject.org/archives/list/kde@lists.fedoraproject.org...
6:25 a.m.
On Tue, Feb 8, 2022 at 9:48 AM Kamil Paral <kparal(a)redhat.com> wrote:
OK. I sent out the complete criterion proposal to Workstation and
KDE
lists:
https://lists.fedoraproject.org/archives/list/desktop@lists.fedoraproject...
https://lists.fedoraproject.org/archives/list/kde@lists.fedoraproject.org...
Because there was no negative feedback on those lists, I made the criterion
live just now:
https://fedoraproject.org/wiki/Fedora_36_Final_Release_Criteria#software-...
Thanks everyone for participating!
10:56 a.m.
On Tue, 2022-02-15 at 13:25 +0100, Kamil Paral wrote:
On Tue, Feb 8, 2022 at 9:48 AM Kamil Paral <kparal(a)redhat.com>
wrote:
> OK. I sent out the complete criterion proposal to Workstation and KDE
> lists:
>
>
https://lists.fedoraproject.org/archives/list/desktop@lists.fedoraproject...
>
>
https://lists.fedoraproject.org/archives/list/kde@lists.fedoraproject.org...
>
>
Because there was no negative feedback on those lists, I made the criterion
live just now:
https://fedoraproject.org/wiki/Fedora_36_Final_Release_Criteria#software-...
Thanks everyone for participating!
Thanks Kamil! So the next step would be to draft test cases for the
additional blocking functionality. Do we have a plan for that?
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
3:53 a.m.
Thanks Kamil! So the next step would be to draft test cases for the
additional blocking functionality. Do we have a plan for that?
I can suggest something as I would like to convert it into OpenQA later, so
I might hit two targets with one arrow.
Luk
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
_______________________________________________
test mailing list -- test(a)lists.fedoraproject.org
To unsubscribe send an email to test-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
--
Lukáš Růžička
FEDORA QE, RHCE
Red Hat
<https://www.redhat.com>
Purkyňova 115
612 45 Brno - Královo Pole
lruzicka(a)redhat.com
TRIED AND PERSONALLY TESTED, ERGO TRUSTED. <https://redhat.com/trusted>
8:37 a.m.
On Thu, Feb 17, 2022 at 10:57 AM Lukas Ruzicka <lruzicka(a)redhat.com> wrote:
> Thanks Kamil! So the next step would be to draft test cases for the
> additional blocking functionality. Do we have a plan for that?
>
I can suggest something as I would like to convert it into OpenQA later, so
I might hit two targets with one arrow.
https://pagure.io/fedora-qa/issue/688
12:13 a.m.
On Fri, Jan 21, 2022 at 3:54 PM Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
The release criteria aren't design documents. We're not defining the
intended behavior, exactly; we're defining a certain subset of known
expected behavior which we require to *work*.
The situations you describe are, essentially, "disaster recovery"
scenarios. What our current package managers do in this scenario is
"leave things in a corrupted state".
Yep. The (outdated, being updated) Workstation product requirements
doc says "Upgrade should be a safe process that never leaves the
system needing manual intervention." And that's not the case if
there's a crash or power failure. It's definitely manual intervention
time to recover.
But how could Fedora QA enforce what amounts to an aspirational
requirement? I think it's a good aspiration and should be in the next
revision, but also needs recommitting to it.
That's been the case for years and
will likely continue to be so.
I hope it won't continue much longer. On the one hand rpm-ostree is
pretty much there, and on the other hand btrfs can make it trivially
cheap to apply updates/upgrades on snapshots *instead* of the current
running system. A crash or powerfail with either transactional model
means you're booting the current unmodified root. So there is a way
forward.
So, I'd say there isn't really anything useful we could do by
writing
criteria around the scenario of interrupted updates.
I think there's a "plug in your laptop" requirement by GNOME Software.
If for some reason that check, and inhibitor, weren't working, i.e. a
regression, I think it could be considered a blocker. It'd be a
manifestly bad idea to let a laptop start doing an major version
upgrade without power. If the system shuts down during package
installation, manual intervention would be required to recover.
It's worth noting that we do have one rpm-ostree based
release-blocking
edition at present - IoT - and we may have more (e.g. Silverblue and/or
CoreOS) in future. For rpm-ostree based installs, we can and do assert
in the release criteria that rollbacks and rebases work, for recovery
from broken update attempts:
Yep, good
https://fedoraproject.org/wiki/Basic_Release_Criteria#rpm-ostree_requirem...
because that is how things are designed to work, and it's clearly
important enough that we should make it part of the release criteria
that things actually work that way. But for traditional RPM-based
installs, it doesn't really seem like something we can usefully address
in the criteria to me.
Not yet.
--
Chris Murphy
4:55 p.m.
On Thu, 2022-01-20 at 17:54 +0100, Kamil Paral wrote:
This is another attempt at creating a new release criterion specific
for
graphical package managers. We already discussed it in November, but we
couldn't agree easily on some particular details, and then I was gone for a
long time:
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.or...
Here is my new attempt. The new criterion text is longer, but I believe it
is overall somewhat weaker (to accommodate some of the previous concerns),
just more explicit. As before, please note that we already have package
management partly covered in the Basic criteria [1] and Beta criteria [2].
Please read those first, including footnotes. The following new criterion
is proposed against the Final milestone:
This definitely seems like an improvement to me. Thanks a lot for
drafting it. I think it's probably fine to go ahead with it at least as
a first take, we can always refine it later as we go along.
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
5:49 a.m.
On Thu, Jan 20, 2022 at 10:26 PM Kamil Paral <kparal(a)redhat.com> wrote:
This is another attempt at creating a new release criterion specific
for
graphical package managers. We already discussed it in November, but we
couldn't agree easily on some particular details, and then I was gone for a
long time:
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.or...
Here is my new attempt. The new criterion text is longer, but I believe it
is overall somewhat weaker (to accommodate some of the previous concerns),
just more explicit. As before, please note that we already have package
management partly covered in the Basic criteria [1] and Beta criteria [2].
Please read those first, including footnotes. The following new criterion
is proposed against the Final milestone:
~~~~~~~~~~~~~~~~~~~~
The default graphical package manager for a given software type must
appropriately:
* Install, remove and update software
* List locally-installed software coming from the official Fedora
repositories
* List available software (possibly in categories, a curated list, etc)
* Display metadata relevant to the selected software (e.g. the
description, screenshots, the size)
* Start the selected installed software
* Configure software sources by enabling/disabling pre-defined official
repositories and then adjust the available software pool accordingly
* Notify the user when system updates are available (taking into account
the intended frequency of such notifications)
The following must hold true:
* When multiple operations (covered by this criterion) are requested, all
of them must finish correctly. (It's also valid to inform the user that a
certain operation is not available at that moment, see the "Supported
functionality and design decisions" note).
* The displayed state of software or software sources must not differ from
their actual state. (E.g. an RPM package must not be shown as installed
when it is not, a repository must not be shown as disabled or missing when
it is enabled, etc).
* Usual GUI interactivity must not break operations covered by this
criterion. (E.g. when the GUI allows the user to click some buttons while
an operation is running, it must not break that operation).
* The package manager must never make the system enter an inconsistent or
unbootable state. (E.g. damage the local software database, remove wrong
system files, break the bootloader, etc).
Note: Supported functionality and design decisions
All requirements apply only if such a functionality is supported by the
package manager; it does not imply that the package manager must support
all this functionality. The requirements don't apply if some functionality
is intentionally modified as a design decision (e.g. if some software
sources can't be disabled as an intentional precaution to users breaking
their system). If there is a bug violating the design decisions in one of
the covered areas (e.g. a software source which is supposed to be always
enabled can be disabled), it's up to the blocker review team to consider
whether this bug makes the user experience or system safety considerably
worse.
Note: Refer to Basic footnotes
The footnotes for the [similar Basic criterion covering console tools][1]
regarding software types, 'appropriate' behavior, scope and so on are all
generally applicable to this criterion also.
~~~~~~~~~~~~~~~~~~~~
If you compare it to the previous proposal, the new one avoids talking
about "sequential or concurrent" operations. That leaves some more wiggle
room when arguing that specific approaches are too niche (Frantisek's
concern), i.e. it no longer clearly covers these approaches completely.
However, it still makes sure that multiple operations are covered at least
at a basic level (which should cover our famous "install and then remove
the same package"). The tool can also say "no can do", and when that
happens to be some internal error message, we can discuss how clear it is
to the user.
I updated the "list installed software" and "configure sources"
requirements with suggestions from Ben.
It also explicitly mentions that just basic clicking through GUI must not
abort running operations (which was something that got good consensus). And
adds an obvious statement about not breaking the system, which was implied
in the past but I assumed it's better to have it clearly visible (this one
could be moved to Beta, possibly).
Finally it adds a requirement that the package manager must not mislead
the user by showing e.g. something as installed when it is not. You can say
this was also implied in the past (otherwise the 'install' or 'list'
operation is not working correctly). but again, this is just to avoid "it
works well, just displays it wrongly" arguments in the future.
What do you think?
[1]
https://fedoraproject.org/wiki/Basic_Release_Criteria#software-install-re...
[2]
https://fedoraproject.org/wiki/Fedora_35_Beta_Release_Criteria#Installing...
_______________________________________________
test mailing list -- test(a)lists.fedoraproject.org
To unsubscribe send an email to test-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
This seems to be an improved version compared to the last one. I think as
adamw mentioned, we should move ahead with it and
update it when the need comes.
--
//sumantro
Fedora QE
TRIED AND PERSONALLY TESTED, ERGO TRUSTED <https://redhat.com/trusted>
797
days inactive
825
days old
24 comments
7 participants
participants (7)
-
Adam Williamson -
Ben Cotton -
Chris Murphy -
John Mellor -
Kamil Paral -
Lukas Ruzicka -
Sumantro Mukherjee