This is another attempt at creating a new release criterion specific for graphical package managers. We already discussed it in November, but we couldn't agree easily on some particular details, and then I was gone for a long time: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/t...
Here is my new attempt. The new criterion text is longer, but I believe it is overall somewhat weaker (to accommodate some of the previous concerns), just more explicit. As before, please note that we already have package management partly covered in the Basic criteria [1] and Beta criteria [2]. Please read those first, including footnotes. The following new criterion is proposed against the Final milestone:
~~~~~~~~~~~~~~~~~~~~ The default graphical package manager for a given software type must appropriately: * Install, remove and update software * List locally-installed software coming from the official Fedora repositories * List available software (possibly in categories, a curated list, etc) * Display metadata relevant to the selected software (e.g. the description, screenshots, the size) * Start the selected installed software * Configure software sources by enabling/disabling pre-defined official repositories and then adjust the available software pool accordingly * Notify the user when system updates are available (taking into account the intended frequency of such notifications)
The following must hold true: * When multiple operations (covered by this criterion) are requested, all of them must finish correctly. (It's also valid to inform the user that a certain operation is not available at that moment, see the "Supported functionality and design decisions" note). * The displayed state of software or software sources must not differ from their actual state. (E.g. an RPM package must not be shown as installed when it is not, a repository must not be shown as disabled or missing when it is enabled, etc). * Usual GUI interactivity must not break operations covered by this criterion. (E.g. when the GUI allows the user to click some buttons while an operation is running, it must not break that operation). * The package manager must never make the system enter an inconsistent or unbootable state. (E.g. damage the local software database, remove wrong system files, break the bootloader, etc).
Note: Supported functionality and design decisions All requirements apply only if such a functionality is supported by the package manager; it does not imply that the package manager must support all this functionality. The requirements don't apply if some functionality is intentionally modified as a design decision (e.g. if some software sources can't be disabled as an intentional precaution to users breaking their system). If there is a bug violating the design decisions in one of the covered areas (e.g. a software source which is supposed to be always enabled can be disabled), it's up to the blocker review team to consider whether this bug makes the user experience or system safety considerably worse.
Note: Refer to Basic footnotes The footnotes for the [similar Basic criterion covering console tools][1] regarding software types, 'appropriate' behavior, scope and so on are all generally applicable to this criterion also. ~~~~~~~~~~~~~~~~~~~~
If you compare it to the previous proposal, the new one avoids talking about "sequential or concurrent" operations. That leaves some more wiggle room when arguing that specific approaches are too niche (Frantisek's concern), i.e. it no longer clearly covers these approaches completely. However, it still makes sure that multiple operations are covered at least at a basic level (which should cover our famous "install and then remove the same package"). The tool can also say "no can do", and when that happens to be some internal error message, we can discuss how clear it is to the user.
I updated the "list installed software" and "configure sources" requirements with suggestions from Ben.
It also explicitly mentions that just basic clicking through GUI must not abort running operations (which was something that got good consensus). And adds an obvious statement about not breaking the system, which was implied in the past but I assumed it's better to have it clearly visible (this one could be moved to Beta, possibly).
Finally it adds a requirement that the package manager must not mislead the user by showing e.g. something as installed when it is not. You can say this was also implied in the past (otherwise the 'install' or 'list' operation is not working correctly). but again, this is just to avoid "it works well, just displays it wrongly" arguments in the future.
What do you think?
[1] https://fedoraproject.org/wiki/Basic_Release_Criteria#software-install-remov... [2] https://fedoraproject.org/wiki/Fedora_35_Beta_Release_Criteria#Installing.2C...
On Thu, Jan 20, 2022 at 9:55 AM Kamil Paral kparal@redhat.com wrote:
This is another attempt at creating a new release criterion specific for graphical package managers. We already discussed it in November, but we couldn't agree easily on some particular details, and then I was gone for a long time: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/t...
Here is my new attempt. The new criterion text is longer, but I believe it is overall somewhat weaker (to accommodate some of the previous concerns), just more explicit. As before, please note that we already have package management partly covered in the Basic criteria [1] and Beta criteria [2]. Please read those first, including footnotes. The following new criterion is proposed against the Final milestone:
The default graphical package manager for a given software type must appropriately: * Install, remove and update software * List locally-installed software coming from the official Fedora repositories * List available software (possibly in categories, a curated list, etc) * Display metadata relevant to the selected software (e.g. the description, screenshots, the size) * Start the selected installed software * Configure software sources by enabling/disabling pre-defined official repositories and then adjust the available software pool accordingly * Notify the user when system updates are available (taking into account the intended frequency of such notifications) The following must hold true: * When multiple operations (covered by this criterion) are requested, all of them must finish correctly. (It's also valid to inform the user that a certain operation is not available at that moment, see the "Supported functionality and design decisions" note). * The displayed state of software or software sources must not differ from their actual state. (E.g. an RPM package must not be shown as installed when it is not, a repository must not be shown as disabled or missing when it is enabled, etc). * Usual GUI interactivity must not break operations covered by this criterion. (E.g. when the GUI allows the user to click some buttons while an operation is running, it must not break that operation). * The package manager must never make the system enter an inconsistent or unbootable state. (E.g. damage the local software database, remove wrong system files, break the bootloader, etc). Note: Supported functionality and design decisions All requirements apply only if such a functionality is supported by the package manager; it does not imply that the package manager must support all this functionality. The requirements don't apply if some functionality is intentionally modified as a design decision (e.g. if some software sources can't be disabled as an intentional precaution to users breaking their system). If there is a bug violating the design decisions in one of the covered areas (e.g. a software source which is supposed to be always enabled can be disabled), it's up to the blocker review team to consider whether this bug makes the user experience or system safety considerably worse. Note: Refer to Basic footnotes The footnotes for the [similar Basic criterion covering console tools][1] regarding software types, 'appropriate' behavior, scope and so on are all generally applicable to this criterion also.If you compare it to the previous proposal, the new one avoids talking about "sequential or concurrent" operations. That leaves some more wiggle room when arguing that specific approaches are too niche (Frantisek's concern), i.e. it no longer clearly covers these approaches completely. However, it still makes sure that multiple operations are covered at least at a basic level (which should cover our famous "install and then remove the same package"). The tool can also say "no can do", and when that happens to be some internal error message, we can discuss how clear it is to the user.
I updated the "list installed software" and "configure sources" requirements with suggestions from Ben.
It also explicitly mentions that just basic clicking through GUI must not abort running operations (which was something that got good consensus). And adds an obvious statement about not breaking the system, which was implied in the past but I assumed it's better to have it clearly visible (this one could be moved to Beta, possibly).
Finally it adds a requirement that the package manager must not mislead the user by showing e.g. something as installed when it is not. You can say this was also implied in the past (otherwise the 'install' or 'list' operation is not working correctly). but again, this is just to avoid "it works well, just displays it wrongly" arguments in the future.
What do you think?
Looks good to me.
On 2022-01-20 11:54 a.m., Kamil Paral wrote:
This is another attempt at creating a new release criterion specific for graphical package managers. We already discussed it in November, but we couldn't agree easily on some particular details, and then I was gone for a long time: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/t...
Here is my new attempt. The new criterion text is longer, but I believe it is overall somewhat weaker (to accommodate some of the previous concerns), just more explicit. As before, please note that we already have package management partly covered in the Basic criteria [1] and Beta criteria [2]. Please read those first, including footnotes. The following new criterion is proposed against the Final milestone:
The default graphical package manager for a given software type must appropriately: * Install, remove and update software * List locally-installed software coming from the official Fedora repositories * List available software (possibly in categories, a curated list, etc) * Display metadata relevant to the selected software (e.g. the description, screenshots, the size) * Start the selected installed software * Configure software sources by enabling/disabling pre-defined official repositories and then adjust the available software pool accordingly * Notify the user when system updates are available (taking into account the intended frequency of such notifications) The following must hold true: * When multiple operations (covered by this criterion) are requested, all of them must finish correctly. (It's also valid to inform the user that a certain operation is not available at that moment, see the "Supported functionality and design decisions" note). * The displayed state of software or software sources must not differ from their actual state. (E.g. an RPM package must not be shown as installed when it is not, a repository must not be shown as disabled or missing when it is enabled, etc). * Usual GUI interactivity must not break operations covered by this criterion. (E.g. when the GUI allows the user to click some buttons while an operation is running, it must not break that operation). * The package manager must never make the system enter an inconsistent or unbootable state. (E.g. damage the local software database, remove wrong system files, break the bootloader, etc). Note: Supported functionality and design decisions All requirements apply only if such a functionality is supported by the package manager; it does not imply that the package manager must support all this functionality. The requirements don't apply if some functionality is intentionally modified as a design decision (e.g. if some software sources can't be disabled as an intentional precaution to users breaking their system). If there is a bug violating the design decisions in one of the covered areas (e.g. a software source which is supposed to be always enabled can be disabled), it's up to the blocker review team to consider whether this bug makes the user experience or system safety considerably worse. Note: Refer to Basic footnotes The footnotes for the [similar Basic criterion covering console tools][1] regarding software types, 'appropriate' behavior, scope and so on are all generally applicable to this criterion also.If you compare it to the previous proposal, the new one avoids talking about "sequential or concurrent" operations. That leaves some more wiggle room when arguing that specific approaches are too niche (Frantisek's concern), i.e. it no longer clearly covers these approaches completely. However, it still makes sure that multiple operations are covered at least at a basic level (which should cover our famous "install and then remove the same package"). The tool can also say "no can do", and when that happens to be some internal error message, we can discuss how clear it is to the user.
I updated the "list installed software" and "configure sources" requirements with suggestions from Ben.
It also explicitly mentions that just basic clicking through GUI must not abort running operations (which was something that got good consensus). And adds an obvious statement about not breaking the system, which was implied in the past but I assumed it's better to have it clearly visible (this one could be moved to Beta, possibly).
Finally it adds a requirement that the package manager must not mislead the user by showing e.g. something as installed when it is not. You can say this was also implied in the past (otherwise the 'install' or 'list' operation is not working correctly). but again, this is just to avoid "it works well, just displays it wrongly" arguments in the future.
What do you think?
[1] https://fedoraproject.org/wiki/Basic_Release_Criteria#software-install-remov... [2] https://fedoraproject.org/wiki/Fedora_35_Beta_Release_Criteria#Installing.2C...
I've done generic, app-specific and custom installers and lots of install environments on a lot of architectures for about 45 years of my 50 years working as a mission-critical developer and maintainer. There a lots of people who do not understand the Unix way of doing things or how to take advantage of the Unix filesystem to do things a lot better than Windows, and try to make installation or upgrades behave more like ancient Windows patches. There are also a growing number of Fedora users who are now discarding the GUI updater in favour of dnf installs, primarily due to the GUI updater having extreme and unnecessary reboot requirements.
The suggested criteria list is pretty generic, and offhand I can't think of a current GUI or CLI installer that does not conform to these requirements.
What this list does NOT cover are the pain points of the current product, such as avoiding reboots. Currently, the Fedora GUI installer always reboots twice, even though 80% of the updates do not require reboots at all, and similar to the Debian-based distros, tools such as the needs-restarting component of the yum-utils package exist to show that shared libs or open files are out-of-date and either app or full system reboots are required to put the system into a supportable state. Unlike other distros, there are certain apps, like parts of the Gnome desktop and Firefox, that do not respect a policy of no API changes in a given release, and these apps tend to break routinely because of this broken development model being considered acceptable. If I look at Ubuntu practices for example, they introduce a needs-rebooting flag that is set when a memory-mapped file is altered, or when an API change is intentionally introduced. However, having multiple versions of shared libs in use at the same time is not an error, but a really good Unix/Linux feature that is not a valid criteria for a reboot. The current GUI installer behaves more like Windows, and does not allow this situation when it most certainly should. Fedora should consider fixing these packages that routinely allow developer errors instead of downloading the resulting problems to the end user, and the installer tools should not be aiding and abetting bad development practices.
Just my humble opinion as a past master of installers or all types and sizes.
--
John Mellor
On Thu, Jan 20, 2022 at 12:18 PM John Mellor jmellor@rogers.com wrote:
I've done generic, app-specific and custom installers and lots of install environments on a lot of architectures for about 45 years of my 50 years working as a mission-critical developer and maintainer. There a lots of people who do not understand the Unix way of doing things or how to take advantage of the Unix filesystem to do things a lot better than Windows, and try to make installation or upgrades behave more like ancient Windows patches. There are also a growing number of Fedora users who are now discarding the GUI updater in favour of dnf installs, primarily due to the GUI updater having extreme and unnecessary reboot requirements.
Maybe, but it's probably off topic for the test@ list. I think instead it belongs on the desktop@ list where it can be discussed by the Workstation working group. The working group is due to revise the product requirements document this month.
The central problem with reboots is, we don't have a sophisticated enough process to know for sure whether the environment is in a known good state. What if the kernel has had an oops or is tainted in a way that could negatively impact the update? The current update strategy is fragile - it's making modifications to the currently running system, and if it were to fail it can leave the system broken in a variety of really inconvenient ways. And frankly, no one wants to take a chance on any of that, hence the first reboot is to ensure clean kernel state, and a limited environment with reduced number of services for performing the update with minimal chance of conflict.
No one has argued reboots are inherently mandatory, just that given the resources, frankly it's cheaper to require the two reboots. There are other ways of updating being explored to reduce the reboot to one. And reduce update frequency so that the compulsory reboots aren't happening more than once every couple weeks, hopefully.
On Thu, 2022-01-20 at 14:17 -0500, John Mellor wrote:
The suggested criteria list is pretty generic, and offhand I can't think of a current GUI or CLI installer that does not conform to these requirements.
Just before Fedora 35 was released, the KDE package manager did not conform to several of them:
https://bugzilla.redhat.com/show_bug.cgi?id=2015809 https://bugzilla.redhat.com/show_bug.cgi?id=2011322 https://bugzilla.redhat.com/show_bug.cgi?id=2015491 https://bugzilla.redhat.com/show_bug.cgi?id=2011291 https://bugzilla.redhat.com/show_bug.cgi?id=2011333
It was largely these and a few other bugs that motivated us to draft these criteria.
As I mentioned in my other mail, the point of the release criteria is to specify a subset of expected functionality which is so important we should block the release if it's not met. It's not *meant* to contain anything novel or unexpected. *Everything* in the release criteria should be "the way it's supposed to work already". What we're doing with the release criteria is saying "if these things *don't* work the way they're intended to, that's bad enough that we shouldn't ship the release".
What this list does NOT cover are the pain points of the current product, such as avoiding reboots.
Yes, because reboots are a part of the way it's supposed to work. It would be wrong for the release criteria to say "there shouldn't be any reboots" if some of the package managers the distribution has chosen to use have chosen to require reboots on updates, and the distribution has decided that it's fine with that (as we have). Release criteria are not tools for advocacy about how we we like the software to work. That should be done elsewhere.
Great work, Kamil. I think this addresses most of the concerns. And while I still don't love the idea of including concurrent actions, the argument is good enough that I'll live with it. I have some pedantic arguments about one of the things that must hold true, but that's something to discuss socially. I'm +1 to the modified proposal.
On 2022-01-20 11:54 a.m., Kamil Paral wrote:
This is another attempt at creating a new release criterion specific for graphical package managers. We already discussed it in November, but we couldn't agree easily on some particular details, and then I was gone for a long time: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/t...
Here is my new attempt. The new criterion text is longer, but I believe it is overall somewhat weaker (to accommodate some of the previous concerns), just more explicit. As before, please note that we already have package management partly covered in the Basic criteria [1] and Beta criteria [2]. Please read those first, including footnotes. The following new criterion is proposed against the Final milestone:
The default graphical package manager for a given software type must appropriately: * Install, remove and update software * List locally-installed software coming from the official Fedora repositories * List available software (possibly in categories, a curated list, etc) * Display metadata relevant to the selected software (e.g. the description, screenshots, the size) * Start the selected installed software * Configure software sources by enabling/disabling pre-defined official repositories and then adjust the available software pool accordingly * Notify the user when system updates are available (taking into account the intended frequency of such notifications) The following must hold true: * When multiple operations (covered by this criterion) are requested, all of them must finish correctly. (It's also valid to inform the user that a certain operation is not available at that moment, see the "Supported functionality and design decisions" note). * The displayed state of software or software sources must not differ from their actual state. (E.g. an RPM package must not be shown as installed when it is not, a repository must not be shown as disabled or missing when it is enabled, etc). * Usual GUI interactivity must not break operations covered by this criterion. (E.g. when the GUI allows the user to click some buttons while an operation is running, it must not break that operation). * The package manager must never make the system enter an inconsistent or unbootable state. (E.g. damage the local software database, remove wrong system files, break the bootloader, etc). Note: Supported functionality and design decisions All requirements apply only if such a functionality is supported by the package manager; it does not imply that the package manager must support all this functionality. The requirements don't apply if some functionality is intentionally modified as a design decision (e.g. if some software sources can't be disabled as an intentional precaution to users breaking their system). If there is a bug violating the design decisions in one of the covered areas (e.g. a software source which is supposed to be always enabled can be disabled), it's up to the blocker review team to consider whether this bug makes the user experience or system safety considerably worse. Note: Refer to Basic footnotes The footnotes for the [similar Basic criterion covering console tools][1] regarding software types, 'appropriate' behavior, scope and so on are all generally applicable to this criterion also.If you compare it to the previous proposal, the new one avoids talking about "sequential or concurrent" operations. That leaves some more wiggle room when arguing that specific approaches are too niche (Frantisek's concern), i.e. it no longer clearly covers these approaches completely. However, it still makes sure that multiple operations are covered at least at a basic level (which should cover our famous "install and then remove the same package"). The tool can also say "no can do", and when that happens to be some internal error message, we can discuss how clear it is to the user.
I updated the "list installed software" and "configure sources" requirements with suggestions from Ben.
It also explicitly mentions that just basic clicking through GUI must not abort running operations (which was something that got good consensus). And adds an obvious statement about not breaking the system, which was implied in the past but I assumed it's better to have it clearly visible (this one could be moved to Beta, possibly).
Finally it adds a requirement that the package manager must not mislead the user by showing e.g. something as installed when it is not. You can say this was also implied in the past (otherwise the 'install' or 'list' operation is not working correctly). but again, this is just to avoid "it works well, just displays it wrongly" arguments in the future.
What do you think?
[1] https://fedoraproject.org/wiki/Basic_Release_Criteria#software-install-remov... [2] https://fedoraproject.org/wiki/Fedora_35_Beta_Release_Criteria#Installing.2C...
I am unclear from this proposal on what is supposed to happen if a package is not completely installed. E.g: a powerdown or other outage causes the post-install script in an installed rpm to not be run. A few packages like the kernel can take ten minutes to install properly, and you may not have enough battery left to get that done when the power goes out. The package and its dependencies are installed, but not initialized correctly. Does it force an installer continuation or redo at boot time for these partial installs, does in uninstall the app and its whole chain of dependencies, or does it just leave things in a corrupted state?
Similarly, if an already-installed package is corrupted and then upgraded, what is the expected handling?
--
John ellor
On Fri, 2022-01-21 at 14:20 -0500, John Mellor wrote:
I am unclear from this proposal on what is supposed to happen if a package is not completely installed. E.g: a powerdown or other outage causes the post-install script in an installed rpm to not be run. A few packages like the kernel can take ten minutes to install properly, and you may not have enough battery left to get that done when the power goes out. The package and its dependencies are installed, but not initialized correctly. Does it force an installer continuation or redo at boot time for these partial installs, does in uninstall the app and its whole chain of dependencies, or does it just leave things in a corrupted state?
The release criteria aren't design documents. We're not defining the intended behavior, exactly; we're defining a certain subset of known expected behavior which we require to *work*.
The situations you describe are, essentially, "disaster recovery" scenarios. What our current package managers do in this scenario is "leave things in a corrupted state". That's been the case for years and will likely continue to be so. It seems kinda pointless to have a release criterion asserting that this is the case, and we do not do 'enforced-product-design-by-release-criteria' (where we make up release criteria that describe how we'd *like* things to behave, and then try and use that as a stick to make the developers make things actually behave that way).
So, I'd say there isn't really anything useful we could do by writing criteria around the scenario of interrupted updates.
It's worth noting that we do have one rpm-ostree based release-blocking edition at present - IoT - and we may have more (e.g. Silverblue and/or CoreOS) in future. For rpm-ostree based installs, we can and do assert in the release criteria that rollbacks and rebases work, for recovery from broken update attempts:
https://fedoraproject.org/wiki/Basic_Release_Criteria#rpm-ostree_requirement...
because that is how things are designed to work, and it's clearly important enough that we should make it part of the release criteria that things actually work that way. But for traditional RPM-based installs, it doesn't really seem like something we can usefully address in the criteria to me.
On 2022-01-21 5:54 p.m., Adam Williamson wrote:
On Fri, 2022-01-21 at 14:20 -0500, John Mellor wrote:
I am unclear from this proposal on what is supposed to happen if a package is not completely installed. E.g: a powerdown or other outage causes the post-install script in an installed rpm to not be run. A few packages like the kernel can take ten minutes to install properly, and you may not have enough battery left to get that done when the power goes out. The package and its dependencies are installed, but not initialized correctly. Does it force an installer continuation or redo at boot time for these partial installs, does in uninstall the app and its whole chain of dependencies, or does it just leave things in a corrupted state?
The release criteria aren't design documents. We're not defining the intended behavior, exactly; we're defining a certain subset of known expected behavior which we require to *work*.
The situations you describe are, essentially, "disaster recovery" scenarios. What our current package managers do in this scenario is "leave things in a corrupted state". That's been the case for years and will likely continue to be so. It seems kinda pointless to have a release criterion asserting that this is the case, and we do not do 'enforced-product-design-by-release-criteria' (where we make up release criteria that describe how we'd *like* things to behave, and then try and use that as a stick to make the developers make things actually behave that way).
So, I'd say there isn't really anything useful we could do by writing criteria around the scenario of interrupted updates.
It's worth noting that we do have one rpm-ostree based release-blocking edition at present - IoT - and we may have more (e.g. Silverblue and/or CoreOS) in future. For rpm-ostree based installs, we can and do assert in the release criteria that rollbacks and rebases work, for recovery from broken update attempts:
https://fedoraproject.org/wiki/Basic_Release_Criteria#rpm-ostree_requirement...
because that is how things are designed to work, and it's clearly important enough that we should make it part of the release criteria that things actually work that way. But for traditional RPM-based installs, it doesn't really seem like something we can usefully address in the criteria to me.
Ok, so would that allowance not violate 2 of the proposed criteria:
1. * The displayed state of software or software sources must not differ from their actual state. (E.g. an RPM package must not be shown as installed when it is not, a repository must not be shown as disabled or missing when it is enabled, etc). 2. * The package manager must never make the system enter an inconsistent or unbootable state. (E.g. damage the local software database, remove wrong system files, break the bootloader, etc).
Or is the purpose of this document just to provide validation criteria for the existing installer behaviour?
--
John Mellor
On Fri, 2022-01-21 at 18:20 -0500, John Mellor wrote:
Ok, so would that allowance not violate 2 of the proposed criteria:
differ from their actual state. (E.g. an RPM package must not be shown as installed when it is not, a repository must not be shown as disabled or missing when it is enabled, etc).
- The displayed state of software or software sources must not
inconsistent or unbootable state. (E.g. damage the local software database, remove wrong system files, break the bootloader, etc).
- The package manager must never make the system enter an
We probably should carve out an allowance there, in fact, yeah. Kamil, what do you think?
Or is the purpose of this document just to provide validation criteria for the existing installer behaviour?
Yeah, as I wrote, that's the point of release criteria: they define the most important parts of existing behaviour. We're not designing *new* desired behaviour here.
On Sat, Jan 22, 2022 at 12:36 AM Adam Williamson adamwill@fedoraproject.org wrote:
On Fri, 2022-01-21 at 18:20 -0500, John Mellor wrote:
Ok, so would that allowance not violate 2 of the proposed criteria:
differ from their actual state. (E.g. an RPM package must not be shown as installed when it is not, a repository must not be shown as disabled or missing when it is enabled, etc).
- The displayed state of software or software sources must not
inconsistent or unbootable state. (E.g. damage the local software database, remove wrong system files, break the bootloader, etc).
- The package manager must never make the system enter an
We probably should carve out an allowance there, in fact, yeah. Kamil, what do you think?
I think that I don't understand what the issue is :-) Can you re-phrase "carve out an allowance" and be more specific?
If this is about power outages, I don't see any problem here. A power outage is not the package manager's failure, and so those criteria don't apply. If this is about working with an already broken system (broken rpmdb or similar), the package manager again didn't cause this state (it didn't "make the system enter an inconsistent state"), and so again that criterion doesn't apply. It could possibly display incorrect state of e.g. rpm packages, if rpmdb is broken (but who can say what the actual state is, when the db is broken?), but the same issue could then apply e.g. to "must install, remove and update software". We could add a footer "these criteria only apply when the system is in a consistent state", but in that case we might need to add such a footer to most of our criteria :-) E.g. a file browser must be able to copy files only when the filesystem is in a consistent state.
I would expect that these cases would be considered a conditional blocker, the condition being that the system was broken before, and therefore waived, because on a broken system we can't guarantee anything.
Or was this about something different?
On Mon, 2022-01-24 at 15:28 +0100, Kamil Paral wrote:
On Sat, Jan 22, 2022 at 12:36 AM Adam Williamson adamwill@fedoraproject.org wrote:
On Fri, 2022-01-21 at 18:20 -0500, John Mellor wrote:
Ok, so would that allowance not violate 2 of the proposed criteria:
differ from their actual state. (E.g. an RPM package must not be shown as installed when it is not, a repository must not be shown as disabled or missing when it is enabled, etc).
- The displayed state of software or software sources must not
inconsistent or unbootable state. (E.g. damage the local software database, remove wrong system files, break the bootloader, etc).
- The package manager must never make the system enter an
We probably should carve out an allowance there, in fact, yeah. Kamil, what do you think?
I think that I don't understand what the issue is :-) Can you re-phrase "carve out an allowance" and be more specific?
If this is about power outages, I don't see any problem here. A power outage is not the package manager's failure, and so those criteria don't apply.
I can see someone proposing it, though. "I lost power in the middle of an update and now the system is unbootable". Whether it was "caused" by the power failure or the package manager is kind of a matter of perspective. I just figured it might be worth a footnote to explicitly specify that such situations aren't covered, for clarity. It's technically true that a system whose power got cut could fail all sorts of criteria, I guess...it just feels more likely that someone might think this criterion means the package manager is meant to handle it, I guess.
On Mon, Jan 24, 2022 at 6:19 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2022-01-24 at 15:28 +0100, Kamil Paral wrote:
If this is about power outages, I don't see any problem here. A power outage is not the package manager's failure, and so those criteria don't apply.
I can see someone proposing it, though. "I lost power in the middle of an update and now the system is unbootable". Whether it was "caused" by the power failure or the package manager is kind of a matter of perspective. I just figured it might be worth a footnote to explicitly specify that such situations aren't covered, for clarity. It's technically true that a system whose power got cut could fail all sorts of criteria, I guess...it just feels more likely that someone might think this criterion means the package manager is meant to handle it, I guess.
OK, so what about adding a note like this:
Note: Systems in an inconsistent state While the package manager must not be the primary cause for breaking a system (unbootable, invalid internal structures, etc), it doesn't have to '''prevent''' these events from happening. So if there's e.g. a power outage during its operation or a package with harmful scriptlets is installed, which breaks the system, this is not the fault of the package manager and the criteria above are not considered violated. Similarly, when the package manager operates on an already broken system (e.g. with an inconsistent rpm database), the correct behavior cannot be guaranteed, and therefore the criteria also don't apply.
On 2022-01-25 05:02, Kamil Paral wrote:
On Mon, Jan 24, 2022 at 6:19 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2022-01-24 at 15:28 +0100, Kamil Paral wrote: > If this is about power outages, I don't see any problem here. A power > outage is not the package manager's failure, and so those criteria don't > apply. I can see someone proposing it, though. "I lost power in the middle of an update and now the system is unbootable". Whether it was "caused" by the power failure or the package manager is kind of a matter of perspective. I just figured it might be worth a footnote to explicitly specify that such situations aren't covered, for clarity. It's technically true that a system whose power got cut could fail all sorts of criteria, I guess...it just feels more likely that someone might think this criterion means the package manager is meant to handle it, I guess.OK, so what about adding a note like this:
Note: Systems in an inconsistent state While the package manager must not be the primary cause for breaking a system (unbootable, invalid internal structures, etc), it doesn't have to '''prevent''' these events from happening. So if there's e.g. a power outage during its operation or a package with harmful scriptlets is installed, which breaks the system, this is not the fault of the package manager and the criteria above are not considered violated. Similarly, when the package manager operates on an already broken system (e.g. with an inconsistent rpm database), the correct behavior cannot be guaranteed, and therefore the criteria also don't apply.
I'm ok with that or something similar, but it does point out the need to fill a large gap in keeping the machine sane when something unexpected happens. Perhaps F36 can expedite btrfs gui and cli tools to roll back to the last known sane state in both the normal and diagnostic images. You know, like the Sun time-slider on the gui and maybe even the Apple time machine functionality.
--
John Mellor
On Tue, Jan 25, 2022 at 3:00 PM John Mellor jmellor@rogers.com wrote:
I'm ok with that or something similar, but it does point out the need to fill a large gap in keeping the machine sane when something unexpected happens. Perhaps F36 can expedite btrfs gui and cli tools to roll back to the last known sane state in both the normal and diagnostic images.
You have to talk to the developers about that, not QA ;-)
The criteria looks good to me, but I agree that it might be a double edged sword to say that *The package manager must never make the system enter an inconsistent or unbootable state* as suggested. An explanatory note is also a good thing to have. However I am not convinced that the wording needs to be as complex as proposed.
What if we said something like:
The package manager must never be the primary cause to damage a previously sane installed system in any way.
On Tue, Jan 25, 2022 at 4:54 PM Kamil Paral kparal@redhat.com wrote:
On Tue, Jan 25, 2022 at 3:00 PM John Mellor jmellor@rogers.com wrote:
I'm ok with that or something similar, but it does point out the need to fill a large gap in keeping the machine sane when something unexpected happens. Perhaps F36 can expedite btrfs gui and cli tools to roll back to the last known sane state in both the normal and diagnostic images.
You have to talk to the developers about that, not QA ;-)
test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Tue, 2022-01-25 at 11:02 +0100, Kamil Paral wrote:
OK, so what about adding a note like this:
Note: Systems in an inconsistent state While the package manager must not be the primary cause for breaking a system (unbootable, invalid internal structures, etc), it doesn't have to '''prevent''' these events from happening. So if there's e.g. a power outage during its operation or a package with harmful scriptlets is installed, which breaks the system, this is not the fault of the package manager and the criteria above are not considered violated. Similarly, when the package manager operates on an already broken system (e.g. with an inconsistent rpm database), the correct behavior cannot be guaranteed, and therefore the criteria also don't apply.
That sounds fine to me. Lukas' is shorter, but yours to me is clearer about the purpose of the note, so I like it more. Thanks!
On Mon, Feb 7, 2022 at 9:05 PM Adam Williamson adamwill@fedoraproject.org wrote:
That sounds fine to me. Lukas' is shorter, but yours to me is clearer about the purpose of the note, so I like it more. Thanks!
OK. I sent out the complete criterion proposal to Workstation and KDE lists: https://lists.fedoraproject.org/archives/list/desktop@lists.fedoraproject.or... https://lists.fedoraproject.org/archives/list/kde@lists.fedoraproject.org/th...
On Tue, Feb 8, 2022 at 9:48 AM Kamil Paral kparal@redhat.com wrote:
OK. I sent out the complete criterion proposal to Workstation and KDE lists:
https://lists.fedoraproject.org/archives/list/desktop@lists.fedoraproject.or...
https://lists.fedoraproject.org/archives/list/kde@lists.fedoraproject.org/th...
Because there was no negative feedback on those lists, I made the criterion live just now: https://fedoraproject.org/wiki/Fedora_36_Final_Release_Criteria#software-ins...
Thanks everyone for participating!
On Tue, 2022-02-15 at 13:25 +0100, Kamil Paral wrote:
On Tue, Feb 8, 2022 at 9:48 AM Kamil Paral kparal@redhat.com wrote:
OK. I sent out the complete criterion proposal to Workstation and KDE lists:
https://lists.fedoraproject.org/archives/list/desktop@lists.fedoraproject.or...
https://lists.fedoraproject.org/archives/list/kde@lists.fedoraproject.org/th...
Because there was no negative feedback on those lists, I made the criterion live just now: https://fedoraproject.org/wiki/Fedora_36_Final_Release_Criteria#software-ins...
Thanks everyone for participating!
Thanks Kamil! So the next step would be to draft test cases for the additional blocking functionality. Do we have a plan for that?
Thanks Kamil! So the next step would be to draft test cases for the additional blocking functionality. Do we have a plan for that?
I can suggest something as I would like to convert it into OpenQA later, so I might hit two targets with one arrow.
Luk
-- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net
test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Thu, Feb 17, 2022 at 10:57 AM Lukas Ruzicka lruzicka@redhat.com wrote:
Thanks Kamil! So the next step would be to draft test cases for the additional blocking functionality. Do we have a plan for that?
I can suggest something as I would like to convert it into OpenQA later, so I might hit two targets with one arrow.
On Fri, Jan 21, 2022 at 3:54 PM Adam Williamson adamwill@fedoraproject.org wrote:
The release criteria aren't design documents. We're not defining the intended behavior, exactly; we're defining a certain subset of known expected behavior which we require to *work*.
The situations you describe are, essentially, "disaster recovery" scenarios. What our current package managers do in this scenario is "leave things in a corrupted state".
Yep. The (outdated, being updated) Workstation product requirements doc says "Upgrade should be a safe process that never leaves the system needing manual intervention." And that's not the case if there's a crash or power failure. It's definitely manual intervention time to recover.
But how could Fedora QA enforce what amounts to an aspirational requirement? I think it's a good aspiration and should be in the next revision, but also needs recommitting to it.
That's been the case for years and will likely continue to be so.
I hope it won't continue much longer. On the one hand rpm-ostree is pretty much there, and on the other hand btrfs can make it trivially cheap to apply updates/upgrades on snapshots *instead* of the current running system. A crash or powerfail with either transactional model means you're booting the current unmodified root. So there is a way forward.
So, I'd say there isn't really anything useful we could do by writing criteria around the scenario of interrupted updates.
I think there's a "plug in your laptop" requirement by GNOME Software. If for some reason that check, and inhibitor, weren't working, i.e. a regression, I think it could be considered a blocker. It'd be a manifestly bad idea to let a laptop start doing an major version upgrade without power. If the system shuts down during package installation, manual intervention would be required to recover.
It's worth noting that we do have one rpm-ostree based release-blocking edition at present - IoT - and we may have more (e.g. Silverblue and/or CoreOS) in future. For rpm-ostree based installs, we can and do assert in the release criteria that rollbacks and rebases work, for recovery from broken update attempts:
Yep, good
https://fedoraproject.org/wiki/Basic_Release_Criteria#rpm-ostree_requirement...
because that is how things are designed to work, and it's clearly important enough that we should make it part of the release criteria that things actually work that way. But for traditional RPM-based installs, it doesn't really seem like something we can usefully address in the criteria to me.
Not yet.
On Thu, 2022-01-20 at 17:54 +0100, Kamil Paral wrote:
This is another attempt at creating a new release criterion specific for graphical package managers. We already discussed it in November, but we couldn't agree easily on some particular details, and then I was gone for a long time: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/t...
Here is my new attempt. The new criterion text is longer, but I believe it is overall somewhat weaker (to accommodate some of the previous concerns), just more explicit. As before, please note that we already have package management partly covered in the Basic criteria [1] and Beta criteria [2]. Please read those first, including footnotes. The following new criterion is proposed against the Final milestone:
This definitely seems like an improvement to me. Thanks a lot for drafting it. I think it's probably fine to go ahead with it at least as a first take, we can always refine it later as we go along.
On Thu, Jan 20, 2022 at 10:26 PM Kamil Paral kparal@redhat.com wrote:
This is another attempt at creating a new release criterion specific for graphical package managers. We already discussed it in November, but we couldn't agree easily on some particular details, and then I was gone for a long time:
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/t...
Here is my new attempt. The new criterion text is longer, but I believe it is overall somewhat weaker (to accommodate some of the previous concerns), just more explicit. As before, please note that we already have package management partly covered in the Basic criteria [1] and Beta criteria [2]. Please read those first, including footnotes. The following new criterion is proposed against the Final milestone:
The default graphical package manager for a given software type must appropriately: * Install, remove and update software * List locally-installed software coming from the official Fedora repositories * List available software (possibly in categories, a curated list, etc) * Display metadata relevant to the selected software (e.g. the description, screenshots, the size) * Start the selected installed software * Configure software sources by enabling/disabling pre-defined official repositories and then adjust the available software pool accordingly * Notify the user when system updates are available (taking into account the intended frequency of such notifications) The following must hold true: * When multiple operations (covered by this criterion) are requested, all of them must finish correctly. (It's also valid to inform the user that a certain operation is not available at that moment, see the "Supported functionality and design decisions" note). * The displayed state of software or software sources must not differ from their actual state. (E.g. an RPM package must not be shown as installed when it is not, a repository must not be shown as disabled or missing when it is enabled, etc). * Usual GUI interactivity must not break operations covered by this criterion. (E.g. when the GUI allows the user to click some buttons while an operation is running, it must not break that operation). * The package manager must never make the system enter an inconsistent or unbootable state. (E.g. damage the local software database, remove wrong system files, break the bootloader, etc). Note: Supported functionality and design decisions All requirements apply only if such a functionality is supported by the package manager; it does not imply that the package manager must support all this functionality. The requirements don't apply if some functionality is intentionally modified as a design decision (e.g. if some software sources can't be disabled as an intentional precaution to users breaking their system). If there is a bug violating the design decisions in one of the covered areas (e.g. a software source which is supposed to be always enabled can be disabled), it's up to the blocker review team to consider whether this bug makes the user experience or system safety considerably worse. Note: Refer to Basic footnotes The footnotes for the [similar Basic criterion covering console tools][1] regarding software types, 'appropriate' behavior, scope and so on are all generally applicable to this criterion also.If you compare it to the previous proposal, the new one avoids talking about "sequential or concurrent" operations. That leaves some more wiggle room when arguing that specific approaches are too niche (Frantisek's concern), i.e. it no longer clearly covers these approaches completely. However, it still makes sure that multiple operations are covered at least at a basic level (which should cover our famous "install and then remove the same package"). The tool can also say "no can do", and when that happens to be some internal error message, we can discuss how clear it is to the user.
I updated the "list installed software" and "configure sources" requirements with suggestions from Ben.
It also explicitly mentions that just basic clicking through GUI must not abort running operations (which was something that got good consensus). And adds an obvious statement about not breaking the system, which was implied in the past but I assumed it's better to have it clearly visible (this one could be moved to Beta, possibly).
Finally it adds a requirement that the package manager must not mislead the user by showing e.g. something as installed when it is not. You can say this was also implied in the past (otherwise the 'install' or 'list' operation is not working correctly). but again, this is just to avoid "it works well, just displays it wrongly" arguments in the future.
What do you think?
[1] https://fedoraproject.org/wiki/Basic_Release_Criteria#software-install-remov... [2] https://fedoraproject.org/wiki/Fedora_35_Beta_Release_Criteria#Installing.2C... _______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
This seems to be an improved version compared to the last one. I think as adamw mentioned, we should move ahead with it and update it when the need comes.
-
Adam Williamson -
Ben Cotton -
Chris Murphy -
John Mellor -
Kamil Paral -
Lukas Ruzicka -
Sumantro Mukherjee