https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
== Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE.
== Owner == * Name: [[User:chrismurphy| Chris Murphy]], [[User:Salimma|Michel Alexandre Salim]], [[User:Ngompa|Neal Gompa]] * Email: bugzilla@colorremedies.com, michel@michel-slm.name, ngompa13@gmail.com
== Detailed Description == === Current location === <pre>/var/lib/rpm</pre>
=== New location === <pre>/usr/lib/sysimage/rpm</pre>
<code>/var/lib/rpm</code> will be a symlink pointing to <code>/usr/lib/sysimage/rpm</code>
Changing the file system layout to accommodate a snapshot+rollback regime is implied, but not required by this proposal. For example, Fedora has long placed `/home` on a separate subvolume (or file system) so it can be isolated from system root. Likewise, it makes sense to isolate `/var/log` and possibly `/var/lib/libvirt/images` so these locations continue to carry forward in time, even if the system root does a rollback.
== Feedback ==
There will be no change to DNF as part of this change proposal. DNF's history will remain in `/var` until DNF 5. Discussion continues about the effect of a snapshot+rollback regime on DNF history. [http://lists.rpm.org/pipermail/rpm-ecosystem/2021-December/000769.html Relocate DNF history to /usr.]
Upstream RPM accept the change, but institutionally don't like the loss or weakening of a [http://lists.rpm.org/pipermail/rpm-ecosystem/2021-December/000781.html very well known location] for the database, and [http://lists.rpm.org/pipermail/rpm-ecosystem/2021-December/000781.html anticipate complaints].
== Benefit to Fedora ==
* The RPM database primarily describes the state of `/usr`. Storing the databases in `/usr` will more easily facilitate OS rollback, without affecting `/var`.
* Helps align Fedora variants with each other ** rpm-ostree based systems (including CoreOS, IoT, Silverblue, Kinoite) already use `/usr/lib/sysimage` for rpmdb.
* Consistency with another RPM-based distro, (open)SUSE has made this change
* Accounts for various snapshot+rollback regimes, i.e. it's a beneficial change whether Btrfs or device-mapper based regimes.
== Scope == * Proposal owners: ** changes in rpm package *** create the new path *** create a symlink for the old path pointing to new path
* Other developers: ** changes in SElinux policy
* Release engineering: [https://pagure.io/releng/issue/10441 #Releng issue 10441] * Policies and guidelines: N/A (not needed for this Change) * Trademark approval: N/A (not needed for this Change) * Alignment with Objectives:
== Upgrade/compatibility impact == Change will be applied to offline upgrades, similar to the RPM sqlite database change. A systemd service will move the rpmdb from /var to /usr, then create a symlink pointing to /usr from /var.
# Create `/usr/lib/sysimage/rpm` (rpm package will do this at preinst) # Create symlinks in `/usr/lib/sysimage/rpm/` pointing to files in `/var/lib/rpm/` (rpm package will do this at preinst) # Change the dbpath in `/usr/lib/rpm/macros` to `/usr/lib/sysimage/rpm` (rpm package will be patched to do this on F36+) # Request rpm rebuild the database (done via systemd service) # Remove `/var/lib/rpm` and create a symlink `/var/lib/rpm` -> `/usr/lib/sysimage/rpm` (done via systemd service)
== How To Test ==
# Perform a new clean install, or upgrade a system # Check that `/var/lib/rpm` is a symlink to `/usr/lib/sysimage/rpm` # Check that `/usr/lib/sysimage/rpm` is populated with at least `rpmdb.sqlite`, possibly also `rpmdb.sqlite-shm` and `rpmdb.sqlite-wal` # Confirm `rpm -q <package>` and/or `rpm -qa` still work
== User Experience ==
* symlink `/var/lib/rpm` -> `/usr/lib/sysimage/rpm`
Otherwise, the change should be invisible to users.
== Dependencies == * `rpm-ostree` probably should make `/usr/share/rpm` a symlink to `/usr/lib/sysimage/rpm`, rather than the reverse as it is currently. * `PackageKit` might use inotify on `/var/lib/rpm` need to check if it does and whether it should be changed or add the additional path
== Contingency Plan == * Contingency mechanism: Revert the change, try again the next Fedora release. * Contingency deadline: Beta freeze * Blocks release? Yes
On Wed, Dec 29, 2021 at 3:03 PM Ben Cotton bcotton@redhat.com wrote:
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
== Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE.
How does this work on RO /usr files systems? I thought data in /usr was supposed to be static/ It works for rpm-ostree because it's updated at tree creation time.
== Owner ==
- Name: [[User:chrismurphy| Chris Murphy]], [[User:Salimma|Michel
Alexandre Salim]], [[User:Ngompa|Neal Gompa]]
- Email: bugzilla@colorremedies.com, michel@michel-slm.name, ngompa13@gmail.com
== Detailed Description == === Current location ===
<pre>/var/lib/rpm</pre>
=== New location ===
<pre>/usr/lib/sysimage/rpm</pre>
<code>/var/lib/rpm</code> will be a symlink pointing to <code>/usr/lib/sysimage/rpm</code>
Changing the file system layout to accommodate a snapshot+rollback regime is implied, but not required by this proposal. For example, Fedora has long placed `/home` on a separate subvolume (or file system) so it can be isolated from system root. Likewise, it makes sense to isolate `/var/log` and possibly `/var/lib/libvirt/images` so these locations continue to carry forward in time, even if the system root does a rollback.
== Feedback ==
There will be no change to DNF as part of this change proposal. DNF's history will remain in `/var` until DNF 5. Discussion continues about the effect of a snapshot+rollback regime on DNF history. [http://lists.rpm.org/pipermail/rpm-ecosystem/2021-December/000769.html Relocate DNF history to /usr.]
Upstream RPM accept the change, but institutionally don't like the loss or weakening of a [http://lists.rpm.org/pipermail/rpm-ecosystem/2021-December/000781.html very well known location] for the database, and [http://lists.rpm.org/pipermail/rpm-ecosystem/2021-December/000781.html anticipate complaints].
== Benefit to Fedora ==
- The RPM database primarily describes the state of `/usr`. Storing
the databases in `/usr` will more easily facilitate OS rollback, without affecting `/var`.
- Helps align Fedora variants with each other
** rpm-ostree based systems (including CoreOS, IoT, Silverblue, Kinoite) already use `/usr/lib/sysimage` for rpmdb.
Consistency with another RPM-based distro, (open)SUSE has made this change
Accounts for various snapshot+rollback regimes, i.e. it's a
beneficial change whether Btrfs or device-mapper based regimes.
== Scope ==
- Proposal owners:
** changes in rpm package *** create the new path *** create a symlink for the old path pointing to new path
- Other developers:
** changes in SElinux policy
- Release engineering: [https://pagure.io/releng/issue/10441 #Releng
issue 10441]
- Policies and guidelines: N/A (not needed for this Change)
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives:
== Upgrade/compatibility impact == Change will be applied to offline upgrades, similar to the RPM sqlite database change. A systemd service will move the rpmdb from /var to /usr, then create a symlink pointing to /usr from /var.
# Create `/usr/lib/sysimage/rpm` (rpm package will do this at preinst) # Create symlinks in `/usr/lib/sysimage/rpm/` pointing to files in `/var/lib/rpm/` (rpm package will do this at preinst) # Change the dbpath in `/usr/lib/rpm/macros` to `/usr/lib/sysimage/rpm` (rpm package will be patched to do this on F36+) # Request rpm rebuild the database (done via systemd service) # Remove `/var/lib/rpm` and create a symlink `/var/lib/rpm` -> `/usr/lib/sysimage/rpm` (done via systemd service)
== How To Test ==
# Perform a new clean install, or upgrade a system # Check that `/var/lib/rpm` is a symlink to `/usr/lib/sysimage/rpm` # Check that `/usr/lib/sysimage/rpm` is populated with at least `rpmdb.sqlite`, possibly also `rpmdb.sqlite-shm` and `rpmdb.sqlite-wal` # Confirm `rpm -q <package>` and/or `rpm -qa` still work
== User Experience ==
- symlink `/var/lib/rpm` -> `/usr/lib/sysimage/rpm`
Otherwise, the change should be invisible to users.
== Dependencies ==
- `rpm-ostree` probably should make `/usr/share/rpm` a symlink to
`/usr/lib/sysimage/rpm`, rather than the reverse as it is currently.
- `PackageKit` might use inotify on `/var/lib/rpm` need to check if it
does and whether it should be changed or add the additional path
== Contingency Plan ==
- Contingency mechanism: Revert the change, try again the next Fedora release.
- Contingency deadline: Beta freeze
- Blocks release? Yes
-- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
I don't see how this is FHS compliant, which in turn would make it non-compliant with Fedora Packaging Guidelines, namely:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_filesystem_layou...
The FHS describes /usr here:
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04.html#purpose18
as "/usr is shareable, read-only data" which clearly does not apply to a database that changes.
While /var is described here:
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05.html#purpose31
as "/var contains variable data files" which appears to exactly describe the RPM database.
At this point somebody will no doubt argue that /usr changes on a package update and that the RPM database is a static definition of the currently installed OS files, but evidence says otherwise:
% ls -l /var/lib/rpm
total 378M
-rw-r--r--. 1 root root 378M Dec 28 16:08 rpmdb.sqlite
-rw-r--r--. 1 root root 32K Dec 29 09:27 rpmdb.sqlite-shm
-rw-r--r--. 1 root root 0 Dec 28 16:08 rpmdb.sqlite-wal
While "Dec 28 16:08" is indeed the last time I updated that machine it seems one of the files has changed more recently - no idea what triggered that but clearly the files are not static between updates.
Tom
On 29/12/2021 15:01, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
== Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE.
== Owner ==
- Name: [[User:chrismurphy| Chris Murphy]], [[User:Salimma|Michel
Alexandre Salim]], [[User:Ngompa|Neal Gompa]]
- Email: bugzilla@colorremedies.com, michel@michel-slm.name, ngompa13@gmail.com
== Detailed Description == === Current location ===
<pre>/var/lib/rpm</pre>
=== New location ===
<pre>/usr/lib/sysimage/rpm</pre>
<code>/var/lib/rpm</code> will be a symlink pointing to <code>/usr/lib/sysimage/rpm</code>
Changing the file system layout to accommodate a snapshot+rollback regime is implied, but not required by this proposal. For example, Fedora has long placed `/home` on a separate subvolume (or file system) so it can be isolated from system root. Likewise, it makes sense to isolate `/var/log` and possibly `/var/lib/libvirt/images` so these locations continue to carry forward in time, even if the system root does a rollback.
== Feedback ==
There will be no change to DNF as part of this change proposal. DNF's history will remain in `/var` until DNF 5. Discussion continues about the effect of a snapshot+rollback regime on DNF history. [http://lists.rpm.org/pipermail/rpm-ecosystem/2021-December/000769.html Relocate DNF history to /usr.]
Upstream RPM accept the change, but institutionally don't like the loss or weakening of a [http://lists.rpm.org/pipermail/rpm-ecosystem/2021-December/000781.html very well known location] for the database, and [http://lists.rpm.org/pipermail/rpm-ecosystem/2021-December/000781.html anticipate complaints].
== Benefit to Fedora ==
- The RPM database primarily describes the state of `/usr`. Storing
the databases in `/usr` will more easily facilitate OS rollback, without affecting `/var`.
- Helps align Fedora variants with each other
** rpm-ostree based systems (including CoreOS, IoT, Silverblue, Kinoite) already use `/usr/lib/sysimage` for rpmdb.
Consistency with another RPM-based distro, (open)SUSE has made this change
Accounts for various snapshot+rollback regimes, i.e. it's a
beneficial change whether Btrfs or device-mapper based regimes.
== Scope ==
- Proposal owners:
** changes in rpm package *** create the new path *** create a symlink for the old path pointing to new path
- Other developers:
** changes in SElinux policy
- Release engineering: [https://pagure.io/releng/issue/10441 #Releng
issue 10441]
- Policies and guidelines: N/A (not needed for this Change)
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives:
== Upgrade/compatibility impact == Change will be applied to offline upgrades, similar to the RPM sqlite database change. A systemd service will move the rpmdb from /var to /usr, then create a symlink pointing to /usr from /var.
# Create `/usr/lib/sysimage/rpm` (rpm package will do this at preinst) # Create symlinks in `/usr/lib/sysimage/rpm/` pointing to files in `/var/lib/rpm/` (rpm package will do this at preinst) # Change the dbpath in `/usr/lib/rpm/macros` to `/usr/lib/sysimage/rpm` (rpm package will be patched to do this on F36+) # Request rpm rebuild the database (done via systemd service) # Remove `/var/lib/rpm` and create a symlink `/var/lib/rpm` -> `/usr/lib/sysimage/rpm` (done via systemd service)
== How To Test ==
# Perform a new clean install, or upgrade a system # Check that `/var/lib/rpm` is a symlink to `/usr/lib/sysimage/rpm` # Check that `/usr/lib/sysimage/rpm` is populated with at least `rpmdb.sqlite`, possibly also `rpmdb.sqlite-shm` and `rpmdb.sqlite-wal` # Confirm `rpm -q <package>` and/or `rpm -qa` still work
== User Experience ==
- symlink `/var/lib/rpm` -> `/usr/lib/sysimage/rpm`
Otherwise, the change should be invisible to users.
== Dependencies ==
- `rpm-ostree` probably should make `/usr/share/rpm` a symlink to
`/usr/lib/sysimage/rpm`, rather than the reverse as it is currently.
- `PackageKit` might use inotify on `/var/lib/rpm` need to check if it
does and whether it should be changed or add the additional path
== Contingency Plan ==
- Contingency mechanism: Revert the change, try again the next Fedora release.
- Contingency deadline: Beta freeze
- Blocks release? Yes
On Wed, 29 Dec 2021 at 10:19, Tom Hughes via devel < devel@lists.fedoraproject.org> wrote:
I don't see how this is FHS compliant, which in turn would make it non-compliant with Fedora Packaging Guidelines, namely:
I am in agreement here and think that this is NOT a change to be made in F36 but needs to be worked through the proper channels of 'upstream'. Get the FHS updated and fixed, work out that the change actually is going to be stuck to in SuSE and not rolled back and then push it to Fedora.
Also please realise that the community can eat only so many changes per release no matter how much you want to otherwise. You can probably get this OR the fs-verity in some future release, but not both in the same release and trying to fight for both is going to mean you get neither. In fact, at a certain point too many changes start making it that any CR with certain names attached will get derailed because people are sure there is some sort of agenda or conspiracy which they have to fight to stop.
On Wed, Dec 29, 2021 at 10:51:44AM -0500, Stephen John Smoogen wrote:
Also please realise that the community can eat only so many changes per release no matter how much you want to otherwise. You can probably get this OR the fs-verity in some future release, but not both in the same release and trying to fight for both is going to mean you get neither. In fact, at a certain point too many changes start making it that any CR with certain names attached will get derailed because people are sure there is some sort of agenda or conspiracy which they have to fight to stop.
I'm not sure I understand this point. Chris is proposing this change, and Neal and I listed because I volunteered to help him implement some of the changes.
And Chris is not on the fs-verity change, and neither is Neal.
I do hope everyone give each other the benefit of the doubt and not jump to assuming there is a secret agenda or conspiracy afoot -- I see this raised against the DIGLIM proposal too, at least once -- but if it helps make it clear that there is no agenda being pushed, I'll take my name off this change so there is no confusion.
Best regards,
On Wed, 29 Dec 2021 at 16:16, Michel Alexandre Salim < salimma@fedoraproject.org> wrote:
On Wed, Dec 29, 2021 at 10:51:44AM -0500, Stephen John Smoogen wrote:
Also please realise that the community can eat only so many changes per release no matter how much you want to otherwise. You can probably get
this
OR the fs-verity in some future release, but not both in the same release and trying to fight for both is going to mean you get neither. In fact,
at
a certain point too many changes start making it that any CR with certain names attached will get derailed because people are sure there is some
sort
of agenda or conspiracy which they have to fight to stop.
I'm not sure I understand this point. Chris is proposing this change, and Neal and I listed because I volunteered to help him implement some of the changes.
And Chris is not on the fs-verity change, and neither is Neal.
My apologies.. I lost track of who was on each of these, and I did not double check before posting. That is my fault 100%. I should have just said I have Change Fatigue with all the proposed changes and have left it at that.
I do hope everyone give each other the benefit of the doubt and not jump to assuming there is a secret agenda or conspiracy afoot -- I see this raised against the DIGLIM proposal too, at least once -- but if it helps make it clear that there is no agenda being pushed, I'll take my name off this change so there is no confusion.
My main issue is that people are seeing agenda's, and it isn't just the usual suspects of people who say NO to many changes. After rereading the changes to make sure I know who is on them, the main problem I am seeing is that there is nothing to sell why the standard Fedora contributor will want any of them in their OS. There is not even a 'Hey we did this already for this rebuild and you can play with it to see if it makes sense for you'. There does not seem to be any community involvement beforehand and nothing for people but to 'react against' the change. Add in the number of them and those reactions become looking for 'hidden' agendas etc.
All that said, this isn't the first time this has happened. It is the reason why various large changes usually require groups to sort of make a sample sausage for people to eat before they have to sit through watching the larger sausage made. Because we are all going to have to do that while any of these changes are working through the grinder.
Best regards,
-- Michel Alexandre Salim profile: https://keyoxide.org/michel@michel-slm.name _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Dec 29, 2021 at 05:06:37PM -0500, Stephen John Smoogen wrote:
On Wed, 29 Dec 2021 at 16:16, Michel Alexandre Salim < salimma@fedoraproject.org> wrote:
All that said, this isn't the first time this has happened. It is the reason why various large changes usually require groups to sort of make a sample sausage for people to eat before they have to sit through watching the larger sausage made. Because we are all going to have to do that while any of these changes are working through the grinder.
That's good feedback, thanks. It can probably be done in both this case and fs-verity:
- for the RPMDB change: have a COPR repo with the patched RPM and the systemd service, with clear instructions how to test
Potential issue: if someone enables this without reading the instructions, and did not create the symlink either manually or via the service, this can break their system. Probably have to name it something like 'rpmdb-move-READ-INSTRUCTIONS-FIRST' or something scary like that
- for fsverity: having a COPR with the plugin and some re-signed packages would probably work, so people can get familiar with it
Best regards,
On Wed, Dec 29, 2021 at 10:51:44AM -0500, Stephen John Smoogen wrote:
F36 but needs to be worked through the proper channels of 'upstream'. Get the FHS updated and fixed, work out that the change actually is going to be
Pretty sure that's a non-starter. FHS 3.0 was released in 2015, and the whole thing has been effectively dead since.
On Wed, 29 Dec 2021 at 21:20, Matthew Miller mattdm@fedoraproject.org wrote:
On Wed, Dec 29, 2021 at 10:51:44AM -0500, Stephen John Smoogen wrote:
F36 but needs to be worked through the proper channels of 'upstream'. Get the FHS updated and fixed, work out that the change actually is going to
be
Pretty sure that's a non-starter. FHS 3.0 was released in 2015, and the whole thing has been effectively dead since.
I agree that the FHS and LSB are practically dead, but it is heavily used in our work. FHS compliance is one of the first things package reviews get dinged on and it doesn't help if we start down the 'Do as I say, not as I do' route. I would say we either need to drop FHS compliance or use an amended copy and work with the SuSE people on making that a shared standard.
On Thu, Dec 30, 2021 at 10:09 AM Stephen John Smoogen smooge@gmail.com wrote:
On Wed, 29 Dec 2021 at 21:20, Matthew Miller mattdm@fedoraproject.org wrote:
On Wed, Dec 29, 2021 at 10:51:44AM -0500, Stephen John Smoogen wrote:
F36 but needs to be worked through the proper channels of 'upstream'. Get the FHS updated and fixed, work out that the change actually is going to be
Pretty sure that's a non-starter. FHS 3.0 was released in 2015, and the whole thing has been effectively dead since.
I agree that the FHS and LSB are practically dead, but it is heavily used in our work. FHS compliance is one of the first things package reviews get dinged on and it doesn't help if we start down the 'Do as I say, not as I do' route. I would say we either need to drop FHS compliance or use an amended copy and work with the SuSE people on making that a shared standard.
That's pretty much what has been going on. I've been working in openSUSE for a few years now to rationalize the FHS configuration in openSUSE to more closely match Fedora. As a result, today the only differences between Red Hat and SUSE distributions are in /usr/lib/sysimage/rpm and usage of /srv in packages. I do not think we'll adopt their usage of /srv anytime soon, but RPM-OSTree variants already are working to move rpmdb from /usr/share/rpm to /usr/lib/sysimage/rpm based on upstream feedback. This Change basically makes it so we use the same rpmdb path regardless of Fedora variant.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Thu, Dec 30, 2021 at 10:12:20AM -0500, Neal Gompa wrote:
say, not as I do' route. I would say we either need to drop FHS compliance or use an amended copy and work with the SuSE people on making that a shared standard.
That's pretty much what has been going on. I've been working in openSUSE for a few years now to rationalize the FHS configuration in openSUSE to more closely match Fedora. As a result, today the only
I'd be supportive of an effort to restart the FHS process with buy-in from openSUSE (and ideally Debian, Arch, FreeBSD...). And probably systemd too, from a plain pragmatic point of view.
I think we'd want to either ask Linux Foundation to split it back from the LSB into its own thing — or else explicitly fork into a new standard.
On Thu, Dec 30, 2021 at 01:14:40PM -0500, Matthew Miller wrote:
I'd be supportive of an effort to restart the FHS process with buy-in from openSUSE (and ideally Debian, Arch, FreeBSD...). And probably systemd too, from a plain pragmatic point of view.
I think we'd want to either ask Linux Foundation to split it back from the LSB into its own thing — or else explicitly fork into a new standard.
Wait, I hit send before finishing my thought, which is: we also need a mechanism which reflects reality rather than an abstract design. Like, sure, if we were going to create the whole thing from scratch now with no history, it probably wouldn't look like it does. And distros (even Fedora) have always just made exceptions. It'd be better to have a working, living standard that makes room for that than one which everyone ignores.
Which is to say: I _don't_ think the process should be to get the FHS restarted and updated and in agreement before moving forward, _especially_ when the proposal is already aligned with what openSUSE is doing, and what we're already doing ourselves with IoT/CoreOS/Kinoite/Silverblue.
On Wed, Dec 29, 2021 at 8:52 AM Stephen John Smoogen smooge@gmail.com wrote:
On Wed, 29 Dec 2021 at 10:19, Tom Hughes via devel devel@lists.fedoraproject.org wrote:
I don't see how this is FHS compliant, which in turn would make it non-compliant with Fedora Packaging Guidelines, namely:
I am in agreement here and think that this is NOT a change to be made in F36 but needs to be worked through the proper channels of 'upstream'. Get the FHS updated and fixed, work out that the change actually is going to be stuck to in SuSE and not rolled back and then push it to Fedora.
It's actually /usr/lib not /usr that applies here. https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s06.html
And it's been worked through proper upstream channels for 4+ years. Location http://lists.rpm.org/pipermail/rpm-maint/2017-October/006722.html FHS question http://lists.rpm.org/pipermail/rpm-maint/2017-October/006697.html
There's a bunch of back and forth throughout. The rpmdb isn't really variable data. It's static data that describes other static data.
On Wed, Dec 29, 2021 at 8:19 AM Tom Hughes via devel devel@lists.fedoraproject.org wrote:
I don't see how this is FHS compliant, which in turn would make it non-compliant with Fedora Packaging Guidelines, namely:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_filesystem_layou...
The FHS describes /usr here:
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04.html#purpose18
as "/usr is shareable, read-only data" which clearly does not apply to a database that changes.
In practice it is read-only data, except when software is being installed or updated. The FHS is a PITA sometimes, but it's not advocating for systems that can't be updated or changed..
On 30/12/2021 07:02, Chris Murphy wrote:
On Wed, Dec 29, 2021 at 8:19 AM Tom Hughes via devel devel@lists.fedoraproject.org wrote:
I don't see how this is FHS compliant, which in turn would make it non-compliant with Fedora Packaging Guidelines, namely:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_filesystem_layou...
The FHS describes /usr here:
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04.html#purpose18as "/usr is shareable, read-only data" which clearly does not apply to a database that changes.
In practice it is read-only data, except when software is being installed or updated. The FHS is a PITA sometimes, but it's not advocating for systems that can't be updated or changed..
As I demonstrated later in my email the contents of /var/lib/rpm do change at other times though.
Tom
On Do, 30.12.21 07:05, Fedora Development ML (devel@lists.fedoraproject.org) wrote:
As I demonstrated later in my email the contents of /var/lib/rpm do change at other times though.
Note that there are other things in /usr that are similar to the RPM db in the "mostly-read-only-but-not-quite" sense, i.e. files that are generated locally during updates, but are static outside of updates.
The kernel modules databases for example, i.e. /lib/modules/`uname -r`/*.dep. Or fontconfig for mime db stuff iirc. ldconfig creates so symlinks. And there's more.
I think having that is even fine. Key must be though that the stuff that is generated locally and stored in /usr is:
a) never updated outside of the update operations b) operates on input that is exclusively from the vendor, too (and not dependent on local admin's config choices)
As long as these rules apply the /usr/ tree remains sharable between multiple systems, and hence it should be fine if some files stored there are not written by RPM itself but by tools invoked by it that operate on data also installed by it.
I think one of those days we should make the first rule explicit btw, and by default overmount /usr/ with a read-only bind mount of itself, that RPM then remounts writable immediately before doing an update and remounts back afterwards. (or alternatively, rpm toggles the r/o bit on the btrfs subvolume if /usr/ is one).
Key point I am trying to make here: having stuff in /usr that is not directly RPM payload is not a new thing, it's a common thing we already have been doing since a long time, and it's OK. I welcome if RPM would put its database in /usr/ as well.
Lennart
-- Lennart Poettering, Berlin
On 12/30/21 09:02, Chris Murphy wrote:
On Wed, Dec 29, 2021 at 8:19 AM Tom Hughes via devel devel@lists.fedoraproject.org wrote:
I don't see how this is FHS compliant, which in turn would make it non-compliant with Fedora Packaging Guidelines, namely:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_filesystem_layou...
The FHS describes /usr here:
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04.html#purpose18as "/usr is shareable, read-only data" which clearly does not apply to a database that changes.
In practice it is read-only data, except when software is being installed or updated. The FHS is a PITA sometimes, but it's not advocating for systems that can't be updated or changed..
The rpmdb has traditionally been like that, but it doesn't mean it will stay that way forever more. There are all manner of currently unimplemented use-cases which would require changing the database outside a direct install/update/erase context. Many of those use-cases are related to files and would fall under "but you need writable fs for that anyway" but not all. Of course it'll always be *mostly* read-only data because of the nature of the data, compared to a general purpose db in /var.
- Panu -
On Mo, 03.01.22 11:57, Panu Matilainen (pmatilai@redhat.com) wrote:
On 12/30/21 09:02, Chris Murphy wrote:
On Wed, Dec 29, 2021 at 8:19 AM Tom Hughes via devel devel@lists.fedoraproject.org wrote:
I don't see how this is FHS compliant, which in turn would make it non-compliant with Fedora Packaging Guidelines, namely:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_filesystem_layou...
The FHS describes /usr here:
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04.html#purpose18as "/usr is shareable, read-only data" which clearly does not apply to a database that changes.
In practice it is read-only data, except when software is being installed or updated. The FHS is a PITA sometimes, but it's not advocating for systems that can't be updated or changed..
The rpmdb has traditionally been like that, but it doesn't mean it will stay that way forever more. There are all manner of currently unimplemented use-cases which would require changing the database outside a direct install/update/erase context. Many of those use-cases are related to files and would fall under "but you need writable fs for that anyway" but not all. Of course it'll always be *mostly* read-only data because of the nature of the data, compared to a general purpose db in /var.
Can you provide an example for such feature requests? i.e. where the rpmdb should be writable even though /usr is assumed to be immutable?
Lennart
-- Lennart Poettering, Berlin
* Lennart Poettering:
Can you provide an example for such feature requests? i.e. where the rpmdb should be writable even though /usr is assumed to be immutable?
Maybe if RPM is used to install software under /opt?
Thanks, Florian
On Mo, 03.01.22 14:15, Florian Weimer (fweimer@redhat.com) wrote:
- Lennart Poettering:
Can you provide an example for such feature requests? i.e. where the rpmdb should be writable even though /usr is assumed to be immutable?
Maybe if RPM is used to install software under /opt?
I'd argue that conceptually /opt and /usr should be seen as the same thing for RPM: i.e. during update time both are writable and outside of updates they are both to be considered read-only from RPM's PoV. Hence it should be fine too put the rpmdb in /usr/ too in that case.
Or with other words: if RPM is used to install something in /opt, it should be fine to require that both /usr and /opt are writable then.
Lennart
-- Lennart Poettering, Berlin
On 1/3/22 15:36, Lennart Poettering wrote:
On Mo, 03.01.22 14:15, Florian Weimer (fweimer@redhat.com) wrote:
- Lennart Poettering:
Can you provide an example for such feature requests? i.e. where the rpmdb should be writable even though /usr is assumed to be immutable?
Maybe if RPM is used to install software under /opt?
I'd argue that conceptually /opt and /usr should be seen as the same thing for RPM: i.e. during update time both are writable and outside of updates they are both to be considered read-only from RPM's PoV. Hence it should be fine too put the rpmdb in /usr/ too in that case.
Or with other words: if RPM is used to install something in /opt, it should be fine to require that both /usr and /opt are writable then.
It does seem a bit controversial to require /usr of all things to be writable to be install into /opt, /etc, /boot and whatnot.
- Panu -
On 1/3/22 14:57, Lennart Poettering wrote:
On Mo, 03.01.22 11:57, Panu Matilainen (pmatilai@redhat.com) wrote:
On 12/30/21 09:02, Chris Murphy wrote:
On Wed, Dec 29, 2021 at 8:19 AM Tom Hughes via devel devel@lists.fedoraproject.org wrote:
I don't see how this is FHS compliant, which in turn would make it non-compliant with Fedora Packaging Guidelines, namely:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_filesystem_layou...
The FHS describes /usr here:
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04.html#purpose18as "/usr is shareable, read-only data" which clearly does not apply to a database that changes.
In practice it is read-only data, except when software is being installed or updated. The FHS is a PITA sometimes, but it's not advocating for systems that can't be updated or changed..
The rpmdb has traditionally been like that, but it doesn't mean it will stay that way forever more. There are all manner of currently unimplemented use-cases which would require changing the database outside a direct install/update/erase context. Many of those use-cases are related to files and would fall under "but you need writable fs for that anyway" but not all. Of course it'll always be *mostly* read-only data because of the nature of the data, compared to a general purpose db in /var.
Can you provide an example for such feature requests? i.e. where the rpmdb should be writable even though /usr is assumed to be immutable?
There seems to be this strange underlying assumption that all packaged content lives in /usr when that's not at all the case. To install a kernel, or a config-only package (under etc), or 3rd party software putting stuff under /opt, or... you need a writable rpmdb. Ditto for 'rpm --import'.
But the kind of thing I had in mind when making that comment initially is eg ability to update file states in the rpmdb to reflect local changes from eg network mounting .. well, the typical example would be /usr but in this case that gets a little strange.
In general, "data set by users" is the common case, whether a policy to flag a given package as unremovable or non-updatable, or add a "reason" (dependency or user-installed), or other annotations.
- Panu -
On Wed, Dec 29, 2021 at 03:17:42PM +0000, Tom Hughes via devel wrote:
At this point somebody will no doubt argue that /usr changes on a package update and that the RPM database is a static definition of the currently installed OS files, but evidence says otherwise:
% ls -l /var/lib/rpm
total 378M
-rw-r--r--. 1 root root 378M Dec 28 16:08 rpmdb.sqlite -rw-r--r--. 1 root root 32K Dec 29 09:27 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Dec 28 16:08 rpmdb.sqlite-wal
While "Dec 28 16:08" is indeed the last time I updated that machine it seems one of the files has changed more recently - no idea what triggered that but clearly the files are not static between updates.
That is a sqlite write-ahead log shared memory file used to coordinate access between concurrent clients. Someone who knows more about the depths of DNF and RPM than me will need to comment, but it looks like `dnf list` touches it -- even though `rpm -qa` doesn't.
On Thu, Dec 30, 2021 at 02:27:04AM -0500, Matthew Miller wrote:
On Wed, Dec 29, 2021 at 03:17:42PM +0000, Tom Hughes via devel wrote:
At this point somebody will no doubt argue that /usr changes on a package update and that the RPM database is a static definition of the currently installed OS files, but evidence says otherwise:
% ls -l /var/lib/rpm
total 378M
-rw-r--r--. 1 root root 378M Dec 28 16:08 rpmdb.sqlite -rw-r--r--. 1 root root 32K Dec 29 09:27 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Dec 28 16:08 rpmdb.sqlite-wal
While "Dec 28 16:08" is indeed the last time I updated that machine it seems one of the files has changed more recently - no idea what triggered that but clearly the files are not static between updates.
That is a sqlite write-ahead log shared memory file used to coordinate access between concurrent clients. Someone who knows more about the depths of DNF and RPM than me will need to comment, but it looks like `dnf list` touches it -- even though `rpm -qa` doesn't.
$ sudo strace -efile dnf list ... openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-wal", O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 4 openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-shm", O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 5 ...
What happens if /var/lib is read-only? Changing (fixing?) this would be a pre-requisite to this proposal, we don't want 'dnf list' to break.
Zbyszek
Zbigniew Jędrzejewski-Szmek wrote on Thu, Dec 30, 2021 at 08:29:54AM +0000:
$ sudo strace -efile dnf list ... openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-wal", O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 4 openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-shm", O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 5 ...
What happens if /var/lib is read-only? Changing (fixing?) this would be a pre-requisite to this proposal, we don't want 'dnf list' to break.
That's easy enough to test. Apparently, of the files already exist the open calls work without problem. If the files don't (e.g. cleared by running 'sqlite3 rpmdb.sqlite vacuum'), then sqlite apparently tries to create them and fails horribly -- but so does rpm -qa:
# rpm -qa error: sqlite failure: CREATE TABLE IF NOT EXISTS 'Packages' (hnum INTEGER PRIMARY KEY AUTOINCREMENT,blob BLOB NOT NULL): unable to open database file error: cannot open Packages index using sqlite - No such file or directory (2) error: cannot open Packages database in /var/lib/rpm
or for that matter plain sqlite3: # sqlite3 rpmdb.sqlite 'select * from Packages;' Error: unable to open database file
So for a read-only filesystem, if the db has wal enabled, the files have to exist. (it's possible to disable WAL mode by running the sqlite 'PRAGMA journal_mode=DELETE;' command when the partition is writable beforehand, but I didn't see how to alter the mode while it is readonly)
Anyway, I don't think vacuum ever runs automatically so the files should always be present and it probably should just work™.
FWIW I've been running with my rpmdb in /usr (through a bind mount) for a few years precisely to keep the rpmdb in sync with the rest of /usr for snapshots -- it's pretty much necessary to run rpm -q on old snapshots and figure which version of what was where easily. Unless btrfs somehow becomes able to tie multiple subvolume snapshots together I think it's a good change.
On Thu, Dec 30, 2021 at 05:53:14PM +0900, Dominique Martinet wrote:
So for a read-only filesystem, if the db has wal enabled, the files have to exist. (it's possible to disable WAL mode by running the sqlite 'PRAGMA journal_mode=DELETE;' command when the partition is writable beforehand, but I didn't see how to alter the mode while it is readonly)
FWIW, Sqlite has guidance here:
https://www.sqlite.org/wal.html#readonly
Older versions of SQLite could not read a WAL-mode database that was read-only. In other words, write access was required in order to read a WAL-mode database. This constraint was relaxed beginning with SQLite version 3.22.0 (2018-01-22).
On newer versions of SQLite, a WAL-mode database on read-only media, or a WAL-mode database that lacks write permission, can still be read as long as one or more of the following conditions are met:
1. The -shm and -wal files already exists and are readable
2. There is write permission on the directory containing the database so that the -shm and -wal files can be created.
3. The database connection is opened using the immutable query parameter.
Even though it is possible to open a read-only WAL-mode database, it is good practice to converted to PRAGMA journal_mode=DELETE prior to burning an SQLite database image onto read-only media.
On 12/30/21 12:22, Matthew Miller wrote:
So for a read-only filesystem, if the db has wal enabled, the files have to exist. (it's possible to disable WAL mode by running the sqlite 'PRAGMA journal_mode=DELETE;' command when the partition is writable beforehand, but I didn't see how to alter the mode while it is readonly)
FWIW, Sqlite has guidance here:
Given that the "shared-memory file contains no persistent content", it seems like "rpmdb.sqlite-shm" could be a symlink to /dev/shm/rpmdb.sqlite-shm", or to some other tmpfs location.
Also: https://www.sqlite.org/walformat.html#the_wal_index_or_shm_file
On Thu, Dec 30, 2021 at 01:41:01PM -0800, Gordon Messmer wrote:
Given that the "shared-memory file contains no persistent content", it seems like "rpmdb.sqlite-shm" could be a symlink to /dev/shm/rpmdb.sqlite-shm", or to some other tmpfs location.
It does seem like maybe it could. I'd definitely like someone who is an expert in area this to figure out if that's right before we try any such thing.
On Sat, Jan 01, 2022 at 12:52:57PM -0500, Matthew Miller wrote:
Given that the "shared-memory file contains no persistent content", it seems like "rpmdb.sqlite-shm" could be a symlink to /dev/shm/rpmdb.sqlite-shm", or to some other tmpfs location.
It does seem like maybe it could. I'd definitely like someone who is an expert in area this to figure out if that's right before we try any such thing.
That said, from https://sqlite.org/walformat.html#the_wal_index_or_shm_file:
Since the content of the shm file does not need to be preserved across a crash, the shm file is never fsync()-ed to disk. In fact, if there were a mechanism by which SQLite could tell the operating system to never persist the shm file to disk but always hold it in cache memory, SQLite would use that mechanism to avoid any unnecessary disk I/O associated with the shm file. However, no such mechanism exists in standard posix.
... sounds promising.
_However_, I'm not sure if this is something best done at the filesystem level (what recreates it at as a symlink if the file is removed?) or in rpm or DNF — or in SQLite itself, if they'd accept something that isn't "standard posix".
On Thu, Dec 30, 2021 at 1:31 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Thu, Dec 30, 2021 at 02:27:04AM -0500, Matthew Miller wrote:
On Wed, Dec 29, 2021 at 03:17:42PM +0000, Tom Hughes via devel wrote:
At this point somebody will no doubt argue that /usr changes on a package update and that the RPM database is a static definition of the currently installed OS files, but evidence says otherwise:
% ls -l /var/lib/rpm
total 378M
-rw-r--r--. 1 root root 378M Dec 28 16:08 rpmdb.sqlite -rw-r--r--. 1 root root 32K Dec 29 09:27 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Dec 28 16:08 rpmdb.sqlite-wal
While "Dec 28 16:08" is indeed the last time I updated that machine it seems one of the files has changed more recently - no idea what triggered that but clearly the files are not static between updates.
That is a sqlite write-ahead log shared memory file used to coordinate access between concurrent clients. Someone who knows more about the depths of DNF and RPM than me will need to comment, but it looks like `dnf list` touches it -- even though `rpm -qa` doesn't.
$ sudo strace -efile dnf list ... openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-wal", O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 4 openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-shm", O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 5 ...
What happens if /var/lib is read-only? Changing (fixing?) this would be a pre-requisite to this proposal, we don't want 'dnf list' to break.
Why should it be a prerequisite? In all Fedora editions and spins with dnf, /usr and /var are read-write. In the case of rpm-ostree based editions and spins, they don't include dnf. I agree dnf should tolerate read-only rpmdb files, but I'm not following the logic leading to it being a prerequisite (must tolerate rather than should tolerate).
On Thu, 30 Dec 2021 at 13:58, Chris Murphy lists@colorremedies.com wrote:
On Thu, Dec 30, 2021 at 1:31 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Thu, Dec 30, 2021 at 02:27:04AM -0500, Matthew Miller wrote:
On Wed, Dec 29, 2021 at 03:17:42PM +0000, Tom Hughes via devel wrote:
At this point somebody will no doubt argue that /usr changes on a package update and that the RPM database is a static definition of the currently installed OS files, but evidence says otherwise:
% ls -l /var/lib/rpm
total 378M
-rw-r--r--. 1 root root 378M Dec 28 16:08 rpmdb.sqlite -rw-r--r--. 1 root root 32K Dec 29 09:27 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Dec 28 16:08 rpmdb.sqlite-wal
While "Dec 28 16:08" is indeed the last time I updated that machine it seems one of the files has changed more recently - no idea what triggered that but clearly the files are not static between updates.
That is a sqlite write-ahead log shared memory file used to coordinate access between concurrent clients. Someone who knows more about the
depths
of DNF and RPM than me will need to comment, but it looks like `dnf
list`
touches it -- even though `rpm -qa` doesn't.
$ sudo strace -efile dnf list ... openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-wal",
O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 4
openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-shm",
O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 5
...
What happens if /var/lib is read-only? Changing (fixing?) this would be a pre-requisite to this proposal, we don't want 'dnf list' to break.
Why should it be a prerequisite? In all Fedora editions and spins with dnf, /usr and /var are read-write. In the case of rpm-ostree based editions and spins, they don't include dnf. I agree dnf should tolerate read-only rpmdb files, but I'm not following the logic leading to it being a prerequisite (must tolerate rather than should tolerate).
I think that at least it needs to be ok'd that they can and will work on it to fix. The dnf team may have other things on their queue that they need to focus on this release so having to redesign their plumbing to deal with moved files may not be possible. that leaves F36 shipped with a broken dnf and no way anyone can update or run anything.
On Thu, Dec 30, 2021 at 2:14 PM Stephen John Smoogen smooge@gmail.com wrote:
On Thu, 30 Dec 2021 at 13:58, Chris Murphy lists@colorremedies.com wrote:
On Thu, Dec 30, 2021 at 1:31 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Thu, Dec 30, 2021 at 02:27:04AM -0500, Matthew Miller wrote:
On Wed, Dec 29, 2021 at 03:17:42PM +0000, Tom Hughes via devel wrote:
At this point somebody will no doubt argue that /usr changes on a package update and that the RPM database is a static definition of the currently installed OS files, but evidence says otherwise:
% ls -l /var/lib/rpm
total 378M
-rw-r--r--. 1 root root 378M Dec 28 16:08 rpmdb.sqlite -rw-r--r--. 1 root root 32K Dec 29 09:27 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Dec 28 16:08 rpmdb.sqlite-wal
While "Dec 28 16:08" is indeed the last time I updated that machine it seems one of the files has changed more recently - no idea what triggered that but clearly the files are not static between updates.
That is a sqlite write-ahead log shared memory file used to coordinate access between concurrent clients. Someone who knows more about the depths of DNF and RPM than me will need to comment, but it looks like `dnf list` touches it -- even though `rpm -qa` doesn't.
$ sudo strace -efile dnf list ... openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-wal", O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 4 openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-shm", O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 5 ...
What happens if /var/lib is read-only? Changing (fixing?) this would be a pre-requisite to this proposal, we don't want 'dnf list' to break.
Why should it be a prerequisite? In all Fedora editions and spins with dnf, /usr and /var are read-write. In the case of rpm-ostree based editions and spins, they don't include dnf. I agree dnf should tolerate read-only rpmdb files, but I'm not following the logic leading to it being a prerequisite (must tolerate rather than should tolerate).
I think that at least it needs to be ok'd that they can and will work on it to fix. The dnf team may have other things on their queue that they need to focus on this release so having to redesign their plumbing to deal with moved files may not be possible. that leaves F36 shipped with a broken dnf and no way anyone can update or run anything.
I've already validated this with DNF on openSUSE. This change was executed in openSUSE for the rpmdb in 2017, and the DNF state information in early 2021. DNF handled both transitions fine, since libsolv gets the rpmdb path from rpm configuration.
For the record, I obviously support this change. Responding to a few threads:
On Wed, Dec 29, 2021, at 10:16 AM, Peter Robinson wrote:
How does this work on RO /usr files systems? I thought data in /usr was supposed to be static/ It works for rpm-ostree because it's updated at tree creation time.
I think you know this but it's worth elaborating on here; rpm-ostree supports client-side layering (and overrides too) and even live updates - and those all operate by default while maintaining `/usr` read-only from the perspective of the rest of the system.
The way this works ultimately is quite simple; the underlying filesystem is writable, we just remount it writable *in a private mount namespace*. So even when you do e.g. `rpm-ostree install -A usbguard`, there is no point where *other* processes can write.
People are focusing a bit too much on "read-only" in this thread - it's more about "lifecycle binding" and versioning the binaries together with metadata about the binaries.
On Thu, Dec 30, 2021, at 1:57 PM, Chris Murphy wrote:
What happens if /var/lib is read-only? Changing (fixing?) this would be a pre-requisite to this proposal, we don't want 'dnf list' to break.
Why would /var/lib be read-only, but /usr be writable?
Why should it be a prerequisite? In all Fedora editions and spins with dnf, /usr and /var are read-write. In the case of rpm-ostree based editions and spins, they don't include dnf.
Remember rpm-ostree links to libdnf, and significant code is hence shared. That's part of the ASCII-art architecture diagram in the docs https://coreos.github.io/rpm-ostree/
The way I'd say this is it aligns "traditional" dnf systems with what rpm-ostree has been doing for many years now, and that will help share *even more* code and logic in the future.
On 29/12/2021 16:01, Ben Cotton wrote:
Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE.
It will break FHS compatibility. /usr must contain read-only data.
On 12/29/21 07:26, Vitaly Zaitsev via devel wrote:
On 29/12/2021 16:01, Ben Cotton wrote:
Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE.
It will break FHS compatibility. /usr must contain read-only data.
If /usr really is read-only, then it probably doesn't matter where the rpmdb is, since packages can't be installed (generally).
Moreover, for systems where /usr is read-only and/or shared (especially stateless systems), having the rpmdb on /usr seems like the most rational place for it, if one expects to be able to use rpm to query the package list. Otherwise, there is an implicit coupling of /usr and /var/lib/rpmdb that requires two mounts rather than one.
On Wed, Dec 29, 2021 at 12:48 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/29/21 07:26, Vitaly Zaitsev via devel wrote:
On 29/12/2021 16:01, Ben Cotton wrote:
Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE.
It will break FHS compatibility. /usr must contain read-only data.
If /usr really is read-only, then it probably doesn't matter where the rpmdb is, since packages can't be installed (generally).
Moreover, for systems where /usr is read-only and/or shared (especially stateless systems), having the rpmdb on /usr seems like the most rational place for it, if one expects to be able to use rpm to query the package list. Otherwise, there is an implicit coupling of /usr and /var/lib/rpmdb that requires two mounts rather than one.
Bingo. It's generally accepted that the RPM database changes when the system state changes. And if the system state is not allowed to change, the rpmdb should not either. The bigger problem is that having the RPM database in /var makes it much harder to correctly implement a boot-to-snapshot scheme for Fedora Linux that reasonably maintains system state properly once /var is carved out. The reason that /var *isn't* carved out by default with our Btrfs configuration is because of the RPM database. Once the RPM database is moved, it becomes possible to split /var into its own subvolume and make it trivial to configure a full boot-to-snapshot system leveraging the technologies we have available to us.
On Wed, 2021-12-29 at 12:57 -0500, Neal Gompa wrote:
On Wed, Dec 29, 2021 at 12:48 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/29/21 07:26, Vitaly Zaitsev via devel wrote:
On 29/12/2021 16:01, Ben Cotton wrote:
Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE.
It will break FHS compatibility. /usr must contain read-only data.
If /usr really is read-only, then it probably doesn't matter where the rpmdb is, since packages can't be installed (generally).
Moreover, for systems where /usr is read-only and/or shared (especially stateless systems), having the rpmdb on /usr seems like the most rational place for it, if one expects to be able to use rpm to query the package list. Otherwise, there is an implicit coupling of /usr and /var/lib/rpmdb that requires two mounts rather than one.
Bingo. It's generally accepted that the RPM database changes when the system state changes. And if the system state is not allowed to change, the rpmdb should not either. The bigger problem is that having the RPM database in /var makes it much harder to correctly implement a boot-to-snapshot scheme for Fedora Linux that reasonably maintains system state properly once /var is carved out.
you just need change to change the default --dbpath of rpm [1] , i.e, I suggest instead change default location of rpm , change the dbpath configuration for the atomic images, is just one idea .
[1] man rpm --dbpath DIRECTORY Use the database in DIRECTORY rather than the default path /var/lib/rpm
The reason that /var *isn't* carved out by default with our Btrfs configuration is because of the RPM database. Once the RPM database is moved, it becomes possible to split /var into its own subvolume and make it trivial to configure a full boot-to-snapshot system leveraging the technologies we have available to us.
-- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Dec 29, 2021 at 12:57:25PM -0500, Neal Gompa wrote:
of the RPM database. Once the RPM database is moved, it becomes possible to split /var into its own subvolume and make it trivial to configure a full boot-to-snapshot system leveraging the technologies we have available to us.
The Benefit to Fedora section alludes to this, but doesn't really spell it out... since this seems like a pretty compelling benefit, it probably should.
On Wed, Dec 29, 2021 at 7:36 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Wed, Dec 29, 2021 at 12:57:25PM -0500, Neal Gompa wrote:
of the RPM database. Once the RPM database is moved, it becomes possible to split /var into its own subvolume and make it trivial to configure a full boot-to-snapshot system leveraging the technologies we have available to us.
The Benefit to Fedora section alludes to this, but doesn't really spell it out... since this seems like a pretty compelling benefit, it probably should.
Added. There's more hurdles to jump, just so no one thinks snapshots and rollbacks are showing up in Fedora 36. There's a quandary with /boot which likewise is married to /usr via the vmlinuz kernel modules. As we snapshot /usr how do we retain vmlinuz such that we can still boot those older snapshots? And then how do we do it considering Boot Loader Spec expects "boot" (or $BOOT) volume to be shared among multiple distros?
Related is the "discoverable sub-volumes specification" discussion, so that we have a way of self-described assembly of systems during startup instead of having to depend on fstab, or any other database for that matter, to know how a system should boot. https://lists.freedesktop.org/archives/systemd-devel/2021-November/047059.ht...
On Wed, 29 Dec 2021 at 12:49, Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/29/21 07:26, Vitaly Zaitsev via devel wrote:
On 29/12/2021 16:01, Ben Cotton wrote:
Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE.
It will break FHS compatibility. /usr must contain read-only data.
If /usr really is read-only, then it probably doesn't matter where the rpmdb is, since packages can't be installed (generally).
The modern day case where /usr is read-only is inside a container and you put an overlay or using some sort of linking to /var which is read-write in case of reboots. To me this is like saying 'move everything into /usr but because its volitile move it back into /var but in a sub-directory from where it was so you can keep an image running.' In this case, this doesn't sound like any savings and more of a headache of why did it corrupt this time.
Looking at my /var/lib in F35, the __db.*** files seems to get written to regularly but I am not sure why. That would probably be something to track down and remove from images afterwards.
Moreover, for systems where /usr is read-only and/or shared (especially stateless systems), having the rpmdb on /usr seems like the most rational place for it, if one expects to be able to use rpm to query the package list. Otherwise, there is an implicit coupling of /usr and /var/lib/rpmdb that requires two mounts rather than one. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On 12/29/21 09:59, Stephen John Smoogen wrote:
The modern day case where /usr is read-only is inside a container and you put an overlay or using some sort of linking to /var which is read-write in case of reboots.
Right, that makes sense.
To me this is like saying 'move everything into /usr but because its volitile move it back into /var but in a sub-directory from where it was so you can keep an image running.' In this case, this doesn't sound like any savings and more of a headache of why did it corrupt this time.
But this doesn't. Why would you need to move the rpmdb? Users probably aren't installing rpm packages in containers at run time (particularly if /usr is read-only); installation typically happens when building the container image, at which point /usr isn't read-only.
On Wed, 29 Dec 2021 at 19:00, Gordon Messmer gordon.messmer@gmail.com wrote: [..]
To me this is like saying 'move everything into /usr but because its volitile move it back into /var but in a sub-directory from where it was so you can keep an image running.' In this case, this doesn't sound like any savings and more of a headache of why did it corrupt this time.
But this doesn't. Why would you need to move the rpmdb? Users probably aren't installing rpm packages in containers at run time (particularly if /usr is read-only); installation typically happens when building the container image, at which point /usr isn't read-only.
This is all because none of the file systems available on Linux invented volumes properties.
+15 years ago on ZFS introduction on Solaris people had exactly the same problems and all those issues have been resolved by introduction of the volume property marking that exact volume needs to be snapshotted and cloned on making new boot env (global or non-global zone as well). That issue is not limited only to rpm database content or generally /var content. In case scenario when new boot env is created and on that new tree would be upgraded majost upgrade of the database engine which on first execution will change format of the database files you want to have some "backup solution" that in case if anything would go wrong you want to be able quickly be able back to original state.
In such cases new upgraded database engine binaries will be on /usr and let's say that database data on /srv/database. To create new boot env in which not only / and /usr will be mounted from from exact clones all what needs to be done in case Solaris or Linux with ZFS would be mark volume mounted in /var that it needs to be snapshotted and cloned during created new instance of the bootable filesystems tree.
In other words trying to solve the multiple volumes mount problem by moving more and more to a single volume (because that is the easiest case) is pointless because it will solve only the rpm database problem but it will not solve issues of mounting multiple volumes. Solving that kind of problems on top of all possible to use on Linux like systems is as well pointless because to solve such thing in civilised way it is necessary to have snapshotable FS which in case of linux for now is only btrfs (optionally zfs as well).
Volulumes properties stored in volume metadata inside storage pools solves much more use cases than only /, /usr and /var separation. From that point of view I really do not understand why btrfs developers resist to implement well known ZFS functionality.
In other words when btrfs will have possibility to possibility to describe a set of volumes which will be necessary to snapshot and clone on making new boot env volumes set SuSE will be rolling back move rpm database back to the /var where it should be.
kloczek
On Wed, 29 Dec 2021 at 13:51, Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/29/21 09:59, Stephen John Smoogen wrote:
The modern day case where /usr is read-only is inside a container and you put an overlay or using some sort of linking to /var which is read-write in case of reboots.
Right, that makes sense.
To me this is like saying 'move everything into /usr but because its volitile move it back into /var but in a sub-directory from where it was so you can keep an image running.' In this case, this doesn't sound like any savings and more of a headache of why did it corrupt this time.
But this doesn't. Why would you need to move the rpmdb? Users probably aren't installing rpm packages in containers at run time (particularly if /usr is read-only); installation typically happens when building the container image, at which point /usr isn't read-only.
Most of the containers I am dealing with are
Grab the base image, Create a layer, and add the images you want, Test and deploy the layered image. Update that image over time.
Theoretically people should build the thing from scratch every time but instead you get someone downloading the base image which they have gotten an OK to use, then adding the stuff they need, and then running with that for YEARS because the person who built the first one left long ago and no one wants to break the paycheck program again.
On Wed, Dec 29, 2021 at 3:36 PM Stephen John Smoogen smooge@gmail.com wrote:
On Wed, 29 Dec 2021 at 13:51, Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/29/21 09:59, Stephen John Smoogen wrote:
The modern day case where /usr is read-only is inside a container and you put an overlay or using some sort of linking to /var which is read-write in case of reboots.
Right, that makes sense.
To me this is like saying 'move everything into /usr but because its volitile move it back into /var but in a sub-directory from where it was so you can keep an image running.' In this case, this doesn't sound like any savings and more of a headache of why did it corrupt this time.
But this doesn't. Why would you need to move the rpmdb? Users probably aren't installing rpm packages in containers at run time (particularly if /usr is read-only); installation typically happens when building the container image, at which point /usr isn't read-only.
Most of the containers I am dealing with are Grab the base image, Create a layer, and add the images you want, Test and deploy the layered image. Update that image over time.
Theoretically people should build the thing from scratch every time but instead you get someone downloading the base image which they have gotten an OK to use, then adding the stuff they need, and then running with that for YEARS because the person who built the first one left long ago and no one wants to break the paycheck program again.
This is a very, very old problem: I was dealing with it with OS images 20 years ago.
Dear Gordon,
Why would you need to move the rpmdb? Users probably aren't installing rpm packages in containers at run time (particularly if /usr is read-only); installation typically happens when building the container image, at which point /usr isn't read-only.
I do actually install RPM package inside containers, but in my case I'm using containers more as short-lived virtual machines for testing than as a deployment mechanism. That said, I don't think that this nullifies your point, as I'm not using a read-only /usr inside these containers.
Best wishes,
Sebastian
On 12/29/21 13:21, Sebastian Crane wrote:
Why would you need to move the rpmdb? Users probably aren't installing rpm packages in containers at run time (particularly if /usr is read-only); installation typically happens when building the container image, at which point /usr isn't read-only.
I do actually install RPM package inside containers, but in my case I'm using containers more as short-lived virtual machines for testing than as a deployment mechanism. That said, I don't think that this nullifies your point, as I'm not using a read-only /usr inside these containers.
Yeah, that was poorly worded. Sorry. What I meant was simply that containers don't seem to be a unique use case. If a container has a writable /usr, then that should be a suitable location for the rpmdb. And if the container's /usr is read-only, then the rpmdb doesn't need to be in /var.
On Wed, Dec 29, 2021 at 12:48 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/29/21 07:26, Vitaly Zaitsev via devel wrote:
On 29/12/2021 16:01, Ben Cotton wrote:
Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE.
It will break FHS compatibility. /usr must contain read-only data.
If /usr really is read-only, then it probably doesn't matter where the rpmdb is, since packages can't be installed (generally).
There are plenty of packages which do not write to /usr/, especially third party add-ons over in /opt/ .
On Wed, Dec 29, 2021 at 02:13:19PM -0500, Nico Kadel-Garcia wrote:
On Wed, Dec 29, 2021 at 12:48 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/29/21 07:26, Vitaly Zaitsev via devel wrote:
On 29/12/2021 16:01, Ben Cotton wrote:
Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE.
It will break FHS compatibility. /usr must contain read-only data.
If /usr really is read-only, then it probably doesn't matter where the rpmdb is, since packages can't be installed (generally).
There are plenty of packages which do not write to /usr/, especially third party add-ons over in /opt/ .
It makes more sense for /var to be in a separate volume from /usr, than for /usr and /opt to be in separate volumes though.
Provided /usr and /opt are in the same volume, then snapshotting and rolling back works as expected.
On 29/12/2021 18:47, Gordon Messmer wrote:
If /usr really is read-only, then it probably doesn't matter where the rpmdb is, since packages can't be installed (generally).
dnf opens these database files for writing, even for the simple `dnf list`.
On Thu, Dec 30, 2021 at 12:00:06PM +0100, Vitaly Zaitsev via devel wrote:
On 29/12/2021 18:47, Gordon Messmer wrote:
If /usr really is read-only, then it probably doesn't matter where the rpmdb is, since packages can't be installed (generally).
dnf opens these database files for writing, even for the simple `dnf list`.
If so this is definitely a bug. (However "dnf list" and "dnf download" seem to work as non-root, so I guess it must fall back to read-only?)
Rich.
On 31/12/2021 12:32, Richard W.M. Jones wrote:
However "dnf list" and "dnf download" seem to work as non-root, so I guess it must fall back to read-only?
Without root and `-C` command-line option it will download all metadata to your $HOME.
On Fri, Dec 31, 2021 at 4:32 AM Richard W.M. Jones rjones@redhat.com wrote:
On Thu, Dec 30, 2021 at 12:00:06PM +0100, Vitaly Zaitsev via devel wrote:
On 29/12/2021 18:47, Gordon Messmer wrote:
If /usr really is read-only, then it probably doesn't matter where the rpmdb is, since packages can't be installed (generally).
dnf opens these database files for writing, even for the simple `dnf list`.
If so this is definitely a bug. (However "dnf list" and "dnf download" seem to work as non-root, so I guess it must fall back to read-only?)
Made a note in the change to investigate this. What I'm seeing with stat are ctime and/or mtime updates of the -shm and -wal files. I'm not sure either should be changing. I'm using noatime, but I expect atime updates probably happen with all these files normally.
On Wednesday, December 29, 2021 12:47:43 PM EST Gordon Messmer wrote:
On 12/29/21 07:26, Vitaly Zaitsev via devel wrote:
On 29/12/2021 16:01, Ben Cotton wrote:
Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE.
It will break FHS compatibility. /usr must contain read-only data.
If /usr really is read-only, then it probably doesn't matter where the rpmdb is, since packages can't be installed (generally).
Not only cases where /usr is readonly, what about the case where /usr is on a SSD because it should not be changing much? I have /var on a spinning disk because by definition it is where things live that change frequently.
-Steve
It doesn't happen me since a long time, but rpmdb can be corrupted, and it must be repaired without altering /usr. For me, rpmdb is variable data so it must be placed at /var/lib according to FHS.
And because my sysadmin OCDs it would be disturbing for me relocating it at /usr :-D
On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
- Other developers:
** changes in SElinux policy
Please can you make sure two bugs are filed against libguestfs and supermin components to track this change (if it happens). We now use librpm to parse the RPM database so I think we're OK from the inspection side, but we do use the RPM database's real location in two places:
[supermin] To test if a package has been installed/upated/removed so that we can rebuild our cache
[libguestfs] To build a "phony" Fedora image for testing with a fake RPM database.
== Upgrade/compatibility impact == Change will be applied to offline upgrades, similar to the RPM sqlite database change. A systemd service will move the rpmdb from /var to /usr, then create a symlink pointing to /usr from /var.
Will the symlink also exist on new installs, or only on upgrades?
Rich.
On Fri, Dec 31, 2021 at 4:30 AM Richard W.M. Jones rjones@redhat.com wrote:
On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
- Other developers:
** changes in SElinux policy
Please can you make sure two bugs are filed against libguestfs and supermin components to track this change (if it happens).
Made a note for this in the scope
Will the symlink also exist on new installs, or only on upgrades?
Both.
On 12/29/21 17:01, Ben Cotton wrote:
Upstream RPM accept the change, but institutionally don't like the loss or weakening of a [http://lists.rpm.org/pipermail/rpm-ecosystem/2021-December/000781.html very well known location] for the database, and [http://lists.rpm.org/pipermail/rpm-ecosystem/2021-December/000781.html anticipate complaints].
Just noticed this in the change text...
To clarify the upstream position: there are no plans to change the upstream default, the level of acceptance is merely that if Fedora wants to change it I'm not going to stand in front of the Change truck.
- Panu -
Does anyone know what /var/lib/rpm-state/gconf is used for? Owning package is GConf2-3.2.6-31.fc35.x86_64
On my Fedora 35 Workstation installation, it's empty. So no obvious conflict with the change proposal, but I'd like to make sure it's not something that if used is going to get mad if there's an rpmdb state change that this location isn't privy to.
-- Chris Murphy
On Mon, Jan 3, 2022 at 1:55 PM Chris Murphy lists@colorremedies.com wrote:
Does anyone know what /var/lib/rpm-state/gconf is used for? Owning package is GConf2-3.2.6-31.fc35.x86_64
OK nevermind. GConf2 is dead upstream, and the only thing I have dragging it in is pdfmod.
On 1/3/22 22:55, Chris Murphy wrote:
Does anyone know what /var/lib/rpm-state/gconf is used for? Owning package is GConf2-3.2.6-31.fc35.x86_64
On my Fedora 35 Workstation installation, it's empty. So no obvious conflict with the change proposal, but I'd like to make sure it's not something that if used is going to get mad if there's an rpmdb state change that this location isn't privy to.
See https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/#_savin..., the /var/lib/rpm-state/ directory is what Fedora recommends for communicating state from one scriptlet to another. Not used or endorsed by rpm itself, and not related to the database at all.
- Panu -
On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton bcotton@redhat.com wrote:
(snip)
== Detailed Description == === Current location ===
<pre>/var/lib/rpm</pre>
=== New location ===
<pre>/usr/lib/sysimage/rpm</pre>
<code>/var/lib/rpm</code> will be a symlink pointing to <code>/usr/lib/sysimage/rpm</code>
I did not find a mention of this in the thread or in the Change proposal, so I'll ask: How do you plan to handle the directory -> symlink replacement on upgrade?
As far as I can tell, those always required special treatment via %pretrans scriptlets or something, and even the method currently recommended by the Packaging Guidelines seems to be broken due to the way dnf / RPM verifies validity of transactions.
Additionally, that "special" handling will probably need to stay in the RPM package's .spec file for years, given that upgrades from Fedora 34 to 36 will need to be supported.
Fabio
On Sat, Jan 8, 2022 at 8:58 AM Fabio Valentini decathorpe@gmail.com wrote:
On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton bcotton@redhat.com wrote:
(snip)
== Detailed Description == === Current location ===
<pre>/var/lib/rpm</pre>
=== New location ===
<pre>/usr/lib/sysimage/rpm</pre>
<code>/var/lib/rpm</code> will be a symlink pointing to <code>/usr/lib/sysimage/rpm</code>
I did not find a mention of this in the thread or in the Change proposal, so I'll ask: How do you plan to handle the directory -> symlink replacement on upgrade?
As far as I can tell, those always required special treatment via %pretrans scriptlets or something, and even the method currently recommended by the Packaging Guidelines seems to be broken due to the way dnf / RPM verifies validity of transactions.
Additionally, that "special" handling will probably need to stay in the RPM package's .spec file for years, given that upgrades from Fedora 34 to 36 will need to be supported.
This is documented in the Change: https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Upgrade.2Fcompatibil...
The part that's probably missing there is that the upgraded package will release ownership of /var/lib/rpm and own the /usr/lib/sysimage/rpm directory.
The configuration will change in the upgrade, and then an rpmdb rebuild on reboot will finalize the transition, as rpmdb rebuilds are done by loading the rpmdb in memory, renaming the directory, recreating it, and re-initializing the database files from memory.
This avoids the pitfalls you've described with the pretrans stuff.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Sat, Jan 8, 2022 at 3:39 PM Neal Gompa ngompa13@gmail.com wrote:
On Sat, Jan 8, 2022 at 8:58 AM Fabio Valentini decathorpe@gmail.com wrote:
On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton bcotton@redhat.com wrote:
(snip)
== Detailed Description == === Current location ===
<pre>/var/lib/rpm</pre>
=== New location ===
<pre>/usr/lib/sysimage/rpm</pre>
<code>/var/lib/rpm</code> will be a symlink pointing to <code>/usr/lib/sysimage/rpm</code>
I did not find a mention of this in the thread or in the Change proposal, so I'll ask: How do you plan to handle the directory -> symlink replacement on upgrade?
As far as I can tell, those always required special treatment via %pretrans scriptlets or something, and even the method currently recommended by the Packaging Guidelines seems to be broken due to the way dnf / RPM verifies validity of transactions.
Additionally, that "special" handling will probably need to stay in the RPM package's .spec file for years, given that upgrades from Fedora 34 to 36 will need to be supported.
This is documented in the Change: https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Upgrade.2Fcompatibil...
The part that's probably missing there is that the upgraded package will release ownership of /var/lib/rpm and own the /usr/lib/sysimage/rpm directory.
The configuration will change in the upgrade, and then an rpmdb rebuild on reboot will finalize the transition, as rpmdb rebuilds are done by loading the rpmdb in memory, renaming the directory, recreating it, and re-initializing the database files from memory.
This avoids the pitfalls you've described with the pretrans stuff.
Oh, great. Looks like I'm ~tired~ and missed that this is in the Change proposal after all ... Thanks for confirming that you found a way to handle upgrades.
Fabio
On Sat, Jan 8, 2022 at 7:53 AM Fabio Valentini decathorpe@gmail.com wrote:
On Sat, Jan 8, 2022 at 3:39 PM Neal Gompa ngompa13@gmail.com wrote:
On Sat, Jan 8, 2022 at 8:58 AM Fabio Valentini decathorpe@gmail.com wrote:
On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton bcotton@redhat.com wrote:
(snip)
== Detailed Description == === Current location ===
<pre>/var/lib/rpm</pre>
=== New location ===
<pre>/usr/lib/sysimage/rpm</pre>
<code>/var/lib/rpm</code> will be a symlink pointing to <code>/usr/lib/sysimage/rpm</code>
I did not find a mention of this in the thread or in the Change proposal, so I'll ask: How do you plan to handle the directory -> symlink replacement on upgrade?
As far as I can tell, those always required special treatment via %pretrans scriptlets or something, and even the method currently recommended by the Packaging Guidelines seems to be broken due to the way dnf / RPM verifies validity of transactions.
Additionally, that "special" handling will probably need to stay in the RPM package's .spec file for years, given that upgrades from Fedora 34 to 36 will need to be supported.
This is documented in the Change: https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Upgrade.2Fcompatibil...
The part that's probably missing there is that the upgraded package will release ownership of /var/lib/rpm and own the /usr/lib/sysimage/rpm directory.
The configuration will change in the upgrade, and then an rpmdb rebuild on reboot will finalize the transition, as rpmdb rebuilds are done by loading the rpmdb in memory, renaming the directory, recreating it, and re-initializing the database files from memory.
This avoids the pitfalls you've described with the pretrans stuff.
Oh, great. Looks like I'm ~tired~ and missed that this is in the Change proposal after all ... Thanks for confirming that you found a way to handle upgrades.
I did a manual proof of concept with the pseudo-sequence in the change proposal on a Fedora Rawhide VM. And it did work, and continues to do both GNOME Software and dnf updates OK. This is a sample size of 1, so there's more work to do to make sure it can be done safely, using the rpm sqlite upgrade as a guide. But I can write up the steps I used, so that anyone can test before we have it wired up.
We have considered not applying the change to upgrades. Strictly speaking the release criterion say we only support upgrades from *clean installs* of the current two releases. But in practice quite a lot of users depend on reliable upgrades for *many* releases, and they get mad when things break even when it's been 10+ releases since they did a clean install. And also, Workstation edition PRD "upgrade process should give a result that is the same as an original install". That is a tall order, but so long as it's safe, it's probably better to apply this change to upgrades. If we run into issues or establish the risk is too high, it's not such a big deal to apply the change to only new clean installs.
On Sat, Jan 8, 2022 at 7:53 PM Chris Murphy lists@colorremedies.com wrote:
On Sat, Jan 8, 2022 at 7:53 AM Fabio Valentini decathorpe@gmail.com wrote:
On Sat, Jan 8, 2022 at 3:39 PM Neal Gompa ngompa13@gmail.com wrote:
On Sat, Jan 8, 2022 at 8:58 AM Fabio Valentini decathorpe@gmail.com wrote:
On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton bcotton@redhat.com wrote:
(snip)
== Detailed Description == === Current location ===
<pre>/var/lib/rpm</pre>
=== New location ===
<pre>/usr/lib/sysimage/rpm</pre>
<code>/var/lib/rpm</code> will be a symlink pointing to <code>/usr/lib/sysimage/rpm</code>
I did not find a mention of this in the thread or in the Change proposal, so I'll ask: How do you plan to handle the directory -> symlink replacement on upgrade?
As far as I can tell, those always required special treatment via %pretrans scriptlets or something, and even the method currently recommended by the Packaging Guidelines seems to be broken due to the way dnf / RPM verifies validity of transactions.
Additionally, that "special" handling will probably need to stay in the RPM package's .spec file for years, given that upgrades from Fedora 34 to 36 will need to be supported.
This is documented in the Change: https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Upgrade.2Fcompatibil...
The part that's probably missing there is that the upgraded package will release ownership of /var/lib/rpm and own the /usr/lib/sysimage/rpm directory.
The configuration will change in the upgrade, and then an rpmdb rebuild on reboot will finalize the transition, as rpmdb rebuilds are done by loading the rpmdb in memory, renaming the directory, recreating it, and re-initializing the database files from memory.
This avoids the pitfalls you've described with the pretrans stuff.
Oh, great. Looks like I'm ~tired~ and missed that this is in the Change proposal after all ... Thanks for confirming that you found a way to handle upgrades.
I did a manual proof of concept with the pseudo-sequence in the change proposal on a Fedora Rawhide VM. And it did work, and continues to do both GNOME Software and dnf updates OK. This is a sample size of 1, so there's more work to do to make sure it can be done safely, using the rpm sqlite upgrade as a guide. But I can write up the steps I used, so that anyone can test before we have it wired up.
We have considered not applying the change to upgrades. Strictly speaking the release criterion say we only support upgrades from *clean installs* of the current two releases. But in practice quite a lot of users depend on reliable upgrades for *many* releases, and they get mad when things break even when it's been 10+ releases since they did a clean install. And also, Workstation edition PRD "upgrade process should give a result that is the same as an original install". That is a tall order, but so long as it's safe, it's probably better to apply this change to upgrades. If we run into issues or establish the risk is too high, it's not such a big deal to apply the change to only new clean installs.
I'm personally concerned that the RPM transactions may not be handled atomically if the /var/lib/rpm/ is migrated in the midst of an RPM update. I'm not personally sure if RPM and Berkeley DB will handle that correctly if there's any issues with the migration, particularly if /usr/ overflows in the process of other bulky updates such as libreoffice. Part of my work involves looking for where multiple things can go wrong at once.
On Sat, Jan 8, 2022 at 8:07 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Sat, Jan 8, 2022 at 7:53 PM Chris Murphy lists@colorremedies.com wrote:
On Sat, Jan 8, 2022 at 7:53 AM Fabio Valentini decathorpe@gmail.com wrote:
On Sat, Jan 8, 2022 at 3:39 PM Neal Gompa ngompa13@gmail.com wrote:
On Sat, Jan 8, 2022 at 8:58 AM Fabio Valentini decathorpe@gmail.com wrote:
On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton bcotton@redhat.com wrote:
(snip)
== Detailed Description == === Current location ===
<pre>/var/lib/rpm</pre>
=== New location ===
<pre>/usr/lib/sysimage/rpm</pre>
<code>/var/lib/rpm</code> will be a symlink pointing to <code>/usr/lib/sysimage/rpm</code>
I did not find a mention of this in the thread or in the Change proposal, so I'll ask: How do you plan to handle the directory -> symlink replacement on upgrade?
As far as I can tell, those always required special treatment via %pretrans scriptlets or something, and even the method currently recommended by the Packaging Guidelines seems to be broken due to the way dnf / RPM verifies validity of transactions.
Additionally, that "special" handling will probably need to stay in the RPM package's .spec file for years, given that upgrades from Fedora 34 to 36 will need to be supported.
This is documented in the Change: https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Upgrade.2Fcompatibil...
The part that's probably missing there is that the upgraded package will release ownership of /var/lib/rpm and own the /usr/lib/sysimage/rpm directory.
The configuration will change in the upgrade, and then an rpmdb rebuild on reboot will finalize the transition, as rpmdb rebuilds are done by loading the rpmdb in memory, renaming the directory, recreating it, and re-initializing the database files from memory.
This avoids the pitfalls you've described with the pretrans stuff.
Oh, great. Looks like I'm ~tired~ and missed that this is in the Change proposal after all ... Thanks for confirming that you found a way to handle upgrades.
I did a manual proof of concept with the pseudo-sequence in the change proposal on a Fedora Rawhide VM. And it did work, and continues to do both GNOME Software and dnf updates OK. This is a sample size of 1, so there's more work to do to make sure it can be done safely, using the rpm sqlite upgrade as a guide. But I can write up the steps I used, so that anyone can test before we have it wired up.
We have considered not applying the change to upgrades. Strictly speaking the release criterion say we only support upgrades from *clean installs* of the current two releases. But in practice quite a lot of users depend on reliable upgrades for *many* releases, and they get mad when things break even when it's been 10+ releases since they did a clean install. And also, Workstation edition PRD "upgrade process should give a result that is the same as an original install". That is a tall order, but so long as it's safe, it's probably better to apply this change to upgrades. If we run into issues or establish the risk is too high, it's not such a big deal to apply the change to only new clean installs.
I'm personally concerned that the RPM transactions may not be handled atomically if the /var/lib/rpm/ is migrated in the midst of an RPM update. I'm not personally sure if RPM and Berkeley DB will handle that correctly if there's any issues with the migration, particularly if /usr/ overflows in the process of other bulky updates such as libreoffice. Part of my work involves looking for where multiple things can go wrong at once.
We're not doing the migration during the transaction itself.
On Sat, Jan 8, 2022 at 6:07 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Sat, Jan 8, 2022 at 7:53 PM Chris Murphy lists@colorremedies.com wrote:
On Sat, Jan 8, 2022 at 7:53 AM Fabio Valentini decathorpe@gmail.com wrote:
On Sat, Jan 8, 2022 at 3:39 PM Neal Gompa ngompa13@gmail.com wrote:
On Sat, Jan 8, 2022 at 8:58 AM Fabio Valentini decathorpe@gmail.com wrote:
On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton bcotton@redhat.com wrote:
(snip)
== Detailed Description == === Current location ===
<pre>/var/lib/rpm</pre>
=== New location ===
<pre>/usr/lib/sysimage/rpm</pre>
<code>/var/lib/rpm</code> will be a symlink pointing to <code>/usr/lib/sysimage/rpm</code>
I did not find a mention of this in the thread or in the Change proposal, so I'll ask: How do you plan to handle the directory -> symlink replacement on upgrade?
As far as I can tell, those always required special treatment via %pretrans scriptlets or something, and even the method currently recommended by the Packaging Guidelines seems to be broken due to the way dnf / RPM verifies validity of transactions.
Additionally, that "special" handling will probably need to stay in the RPM package's .spec file for years, given that upgrades from Fedora 34 to 36 will need to be supported.
This is documented in the Change: https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Upgrade.2Fcompatibil...
The part that's probably missing there is that the upgraded package will release ownership of /var/lib/rpm and own the /usr/lib/sysimage/rpm directory.
The configuration will change in the upgrade, and then an rpmdb rebuild on reboot will finalize the transition, as rpmdb rebuilds are done by loading the rpmdb in memory, renaming the directory, recreating it, and re-initializing the database files from memory.
This avoids the pitfalls you've described with the pretrans stuff.
Oh, great. Looks like I'm ~tired~ and missed that this is in the Change proposal after all ... Thanks for confirming that you found a way to handle upgrades.
I did a manual proof of concept with the pseudo-sequence in the change proposal on a Fedora Rawhide VM. And it did work, and continues to do both GNOME Software and dnf updates OK. This is a sample size of 1, so there's more work to do to make sure it can be done safely, using the rpm sqlite upgrade as a guide. But I can write up the steps I used, so that anyone can test before we have it wired up.
We have considered not applying the change to upgrades. Strictly speaking the release criterion say we only support upgrades from *clean installs* of the current two releases. But in practice quite a lot of users depend on reliable upgrades for *many* releases, and they get mad when things break even when it's been 10+ releases since they did a clean install. And also, Workstation edition PRD "upgrade process should give a result that is the same as an original install". That is a tall order, but so long as it's safe, it's probably better to apply this change to upgrades. If we run into issues or establish the risk is too high, it's not such a big deal to apply the change to only new clean installs.
I'm personally concerned that the RPM transactions may not be handled atomically if the /var/lib/rpm/ is migrated in the midst of an RPM update. I'm not personally sure if RPM and Berkeley DB will handle that correctly if there's any issues with the migration, particularly if /usr/ overflows in the process of other bulky updates such as libreoffice. Part of my work involves looking for where multiple things can go wrong at once.
Sqlite applies here, not bdb.
But also note from that change, the rpmdb conversion to sqlite was not done during the system upgrade, but by a systemd unit. https://fedoraproject.org/wiki/Changes/Sqlite_Rpmdb
Same for this change proposal.
That addresses my concern. I'd assumed that it would occur as part of an RPM update, rather than as a triggered out of band event. It makes rebooting mandatory and slightly risky, especially if /usr overflows halfway through, the process, but that becomes far, far less likely with systemd doing it at boot time running at the same time other RPM updates. I do hope the scripting will exit gracefully if /usr overflows or is unwriteable for other reasons, such as being part of the "immutable" Linux concept.
Nico Kadel-Garcia
On Sat, Jan 8, 2022 at 8:30 PM Chris Murphy lists@colorremedies.com wrote:
On Sat, Jan 8, 2022 at 6:07 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Sat, Jan 8, 2022 at 7:53 PM Chris Murphy lists@colorremedies.com wrote:
On Sat, Jan 8, 2022 at 7:53 AM Fabio Valentini decathorpe@gmail.com wrote:
On Sat, Jan 8, 2022 at 3:39 PM Neal Gompa ngompa13@gmail.com wrote:
On Sat, Jan 8, 2022 at 8:58 AM Fabio Valentini decathorpe@gmail.com wrote:
On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton bcotton@redhat.com wrote: >
(snip)
> > == Detailed Description == > === Current location === > <pre>/var/lib/rpm</pre> > > === New location === > <pre>/usr/lib/sysimage/rpm</pre> > > <code>/var/lib/rpm</code> will be a symlink pointing to > <code>/usr/lib/sysimage/rpm</code>
I did not find a mention of this in the thread or in the Change proposal, so I'll ask: How do you plan to handle the directory -> symlink replacement on upgrade?
As far as I can tell, those always required special treatment via %pretrans scriptlets or something, and even the method currently recommended by the Packaging Guidelines seems to be broken due to the way dnf / RPM verifies validity of transactions.
Additionally, that "special" handling will probably need to stay in the RPM package's .spec file for years, given that upgrades from Fedora 34 to 36 will need to be supported.
This is documented in the Change: https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Upgrade.2Fcompatibil...
The part that's probably missing there is that the upgraded package will release ownership of /var/lib/rpm and own the /usr/lib/sysimage/rpm directory.
The configuration will change in the upgrade, and then an rpmdb rebuild on reboot will finalize the transition, as rpmdb rebuilds are done by loading the rpmdb in memory, renaming the directory, recreating it, and re-initializing the database files from memory.
This avoids the pitfalls you've described with the pretrans stuff.
Oh, great. Looks like I'm ~tired~ and missed that this is in the Change proposal after all ... Thanks for confirming that you found a way to handle upgrades.
I did a manual proof of concept with the pseudo-sequence in the change proposal on a Fedora Rawhide VM. And it did work, and continues to do both GNOME Software and dnf updates OK. This is a sample size of 1, so there's more work to do to make sure it can be done safely, using the rpm sqlite upgrade as a guide. But I can write up the steps I used, so that anyone can test before we have it wired up.
We have considered not applying the change to upgrades. Strictly speaking the release criterion say we only support upgrades from *clean installs* of the current two releases. But in practice quite a lot of users depend on reliable upgrades for *many* releases, and they get mad when things break even when it's been 10+ releases since they did a clean install. And also, Workstation edition PRD "upgrade process should give a result that is the same as an original install". That is a tall order, but so long as it's safe, it's probably better to apply this change to upgrades. If we run into issues or establish the risk is too high, it's not such a big deal to apply the change to only new clean installs.
I'm personally concerned that the RPM transactions may not be handled atomically if the /var/lib/rpm/ is migrated in the midst of an RPM update. I'm not personally sure if RPM and Berkeley DB will handle that correctly if there's any issues with the migration, particularly if /usr/ overflows in the process of other bulky updates such as libreoffice. Part of my work involves looking for where multiple things can go wrong at once.
Sqlite applies here, not bdb.
But also note from that change, the rpmdb conversion to sqlite was not done during the system upgrade, but by a systemd unit. https://fedoraproject.org/wiki/Changes/Sqlite_Rpmdb
Same for this change proposal.
-- Chris Murphy _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
== Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE. [snip]
Moving the RPM database to /usr feels incorrect to me, but we should move it to gain the improvements as noted in the feature proposal.
Numerous followups have noted the requirement that /usr contain read-only content, that it be shareable across hosts, and similar concepts. While this may or may not doable now like we could in the past, the bigger thing to me is around the understanding of what /usr contains. It is generally understood that /usr contains [most of] the installed system. What I think is a bigger requirement or expection now is that one can tar up /usr and transport it to another system or virtual machine or container and expect that it will _probably_ work maybe with a bit of tinkering. This is a really valuable thing to have for developers. Moving the RPM database to this tree adds a component that is unnecessary and sort of out of place.
"OK, but you can do tar -X"
The tar example was simply that, an example. I feel the categorization of system data is more important here. Panu brought this up on the referenced RPM mailing list thread years ago. The original /var location for the RPM database was fine, but we're at a point where we kind of have multiple categories of things historically found in /var.
Reading comments and talking to people, the long standing understanding of /var is still "that's stuff you can rm -rf and the system will still work fine". Technically you could remove the RPM database and the system still function, but arguably would still be broken since you really want the RPM database. This use case of removing the RPM database and still having a functioning system is really only useful for data recovery scenarios. You're ultimately going to reinstall. Probably.
So maybe the question is more "what is the correct location for data like the RPM database?" First, how is that data different from the rest of /var? It is host-specific, it's updated by tools we run all the time, and it's stateful. Losing it would render the system not really useful, though the system would still function. Am I missing anything here?
As Lennart noted, we have lots of examples in /usr of changing data. But I'd say the difference between those examples (library symlinks, cache files, etc) and the RPM database is that the loss of something like a library symlink or cache file can be repaired easily but if you lose the RPM database, the system is in an unrepairable state.
As another example... If you use Bitlbee, this would be like losing your account XML file in /var/lib/bitlbee. Sure, bitlbee still works, but that file is important for Bitlbee to work for you. You have to remember to hang on to that if you wipe /var or reinstall or whatever. I'd say the Bitlbee files in /var/lib/bitlbee also fit this slightly different /var data definition, just as an example.
"But what about the FHS?"
Ah, yes, the FHS. So, I am a fan of the FHS. I actually don't care that it doesn't change every week and is relatively stable. Nothing in the FHS prevents the addition of new top level directories.
I would prefer we steer this conversation in the direction of determining a new top level location to store data that fits this category of "stateful but variable".
/srv was introduced to provide a consistent location for data in this category for server daemons (except mail).
/media was introduced to provide a consistent location for removable media mount points since distributions all did things slightly differently.
/run was introduced for what was traditionally in /var/run.
"So what are you suggesting?"
I would like to see Fedora introduce a new top-level directory called:
/state
That holds the RPM database and other variable and stateful data. This keeps it out of the /usr tree and can serve as a location for future data in this category.
Am 10.01.2022 um 17:19 schrieb David Cantrell dcantrell@redhat.com:
On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
== Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE. [snip]
Moving the RPM database to /usr feels incorrect to me, but we should move it to gain the improvements as noted in the feature proposal.
... "But what about the FHS?"
Ah, yes, the FHS. So, I am a fan of the FHS. I actually don't care that it doesn't change every week and is relatively stable. Nothing in the FHS prevents the addition of new top level directories.
I would prefer we steer this conversation in the direction of determining a new top level location to store data that fits this category of "stateful but variable".
/srv was introduced to provide a consistent location for data in this category for server daemons (except mail).
/media was introduced to provide a consistent location for removable media mount points since distributions all did things slightly differently.
/run was introduced for what was traditionally in /var/run.
"So what are you suggesting?"
I would like to see Fedora introduce a new top-level directory called:
/state
That holds the RPM database and other variable and stateful data. This keeps it out of the /usr tree and can serve as a location for future data in this category.
I would like to second David Cantrell's suggestion.
For Fedora Server it seems to me essential to strictly follow FHS. And from an "enterprise" perspective, it is essential that FHS indeed does not change every week.
A /state directory is not included in FHS, sure. But at least it does not blatantly violate existing and accepted regulations.
And the fact that rpm-ostree based installations have already made this change is not a relevant argument for FHS based installations.
Best Peter
David Cantrell dcantrell@redhat.com writes:
Reading comments and talking to people, the long standing understanding of /var is still "that's stuff you can rm -rf and the system will still work fine". Technically you could remove the RPM database and the system still function [...]
Considering that many other valuable mutable state are put under /var are put there, databases, container images, etc. etc. etc, this understanding cannot be correct. We could just leave /var alone.
# sudo du -s /var/lib/*
- FChE
On Mon, Jan 10, 2022 at 12:30 PM Frank Ch. Eigler fche@redhat.com wrote:
David Cantrell dcantrell@redhat.com writes:
Reading comments and talking to people, the long standing understanding of /var is still "that's stuff you can rm -rf and the system will still work fine". Technically you could remove the RPM database and the system still function [...]
Considering that many other valuable mutable state are put under /var are put there, databases, container images, etc. etc. etc, this understanding cannot be correct. We could just leave /var alone.
# sudo du -s /var/lib/*
I *think* what David is suggesting here is that while databases and container images are indeed changeable data that lives in /var (and are likely VERY important to the user), they are not critical to the functioning of the operating system itself. The OS will still boot and you can still log in[1] if /var is wiped clean, though the system will probably not be doing anything useful at this point. Whereas if the RPM database is damaged or removed, a core function of the operating system (installation and updates) will no longer work.
I think there is definite value in considering adding a new location (though it need not be at the top-level) for data like this.
I'll note that the FHS states the following[2] in regards to /var: "Everything that once went into /usr that is written to during system operation (as opposed to installation and software maintenance) must be in /var."
This exception seems to me to be clearly implying that "installation and software maintenance" do not need to be in /var (and as CoreOS has shown, there's value to keeping it somewhere else).
A further quote from the same section of the FHS suggests the use of /usr/var, which is a location we are not currently using in the Fedora Project, but seems like it would fit the requirements for both CoreOS and traditional RPM Fedora. This same hierarchy could be used for /var/log and /var/lib/libvirt/images as well.
[1] With the possible exception of enterprise logins, since SSSD stores its offline cache in /var/lib/sss/db [2] https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05.html#purpose31
On Mon, Jan 10, 2022 at 10:55 AM Stephen Gallagher sgallagh@redhat.com wrote:
A further quote from the same section of the FHS suggests the use of /usr/var, which is a location we are not currently using in the Fedora Project, but seems like it would fit the requirements for both CoreOS and traditional RPM Fedora. This same hierarchy could be used for /var/log and /var/lib/libvirt/images as well.
This is a nice fit /var/lib/rpm -> /usr/var/lib/rpm
But it's an ask directed at those who have already implemented this change, if it's compelling enough for them to change again.
On Mon, Jan 10, 2022 at 5:39 PM Chris Murphy lists@colorremedies.com wrote:
On Mon, Jan 10, 2022 at 10:55 AM Stephen Gallagher sgallagh@redhat.com wrote:
A further quote from the same section of the FHS suggests the use of /usr/var, which is a location we are not currently using in the Fedora Project, but seems like it would fit the requirements for both CoreOS and traditional RPM Fedora. This same hierarchy could be used for /var/log and /var/lib/libvirt/images as well.
This is a nice fit /var/lib/rpm -> /usr/var/lib/rpm
But it's an ask directed at those who have already implemented this change, if it's compelling enough for them to change again.
Sure, but my reasoning is that such an adaptation could be useful for multiple cases, not just the rpmdb case.
On Tue, Jan 11, 2022 at 6:56 AM Stephen Gallagher sgallagh@redhat.com wrote:
On Mon, Jan 10, 2022 at 5:39 PM Chris Murphy lists@colorremedies.com wrote:
On Mon, Jan 10, 2022 at 10:55 AM Stephen Gallagher sgallagh@redhat.com wrote:
A further quote from the same section of the FHS suggests the use of /usr/var, which is a location we are not currently using in the Fedora Project, but seems like it would fit the requirements for both CoreOS and traditional RPM Fedora. This same hierarchy could be used for /var/log and /var/lib/libvirt/images as well.
This is a nice fit /var/lib/rpm -> /usr/var/lib/rpm
But it's an ask directed at those who have already implemented this change, if it's compelling enough for them to change again.
Sure, but my reasoning is that such an adaptation could be useful for multiple cases, not just the rpmdb case.
The multiple cases is exactly what resulted in convergence on /usr/lib/sysimage, ergo
/usr/lib/sysimage/rpm /usr/lib/sysimage/dpkg /usr/lib/sysimage/pacman /usr/lib/sysimage/dnf
-- Chris Murphy
On Mon, Jan 10, 2022 at 12:55:03PM -0500, Stephen Gallagher wrote:
On Mon, Jan 10, 2022 at 12:30 PM Frank Ch. Eigler fche@redhat.com wrote:
David Cantrell dcantrell@redhat.com writes:
Reading comments and talking to people, the long standing understanding of /var is still "that's stuff you can rm -rf and the system will still work fine". Technically you could remove the RPM database and the system still function [...]
Considering that many other valuable mutable state are put under /var are put there, databases, container images, etc. etc. etc, this understanding cannot be correct. We could just leave /var alone.
# sudo du -s /var/lib/*
I *think* what David is suggesting here is that while databases and container images are indeed changeable data that lives in /var (and are likely VERY important to the user), they are not critical to the functioning of the operating system itself. The OS will still boot and you can still log in[1] if /var is wiped clean, though the system will probably not be doing anything useful at this point. Whereas if the RPM database is damaged or removed, a core function of the operating system (installation and updates) will no longer work.
Yes, that is what I was suggesting.
I think there is definite value in considering adding a new location (though it need not be at the top-level) for data like this.
I'll note that the FHS states the following[2] in regards to /var: "Everything that once went into /usr that is written to during system operation (as opposed to installation and software maintenance) must be in /var."
This exception seems to me to be clearly implying that "installation and software maintenance" do not need to be in /var (and as CoreOS has shown, there's value to keeping it somewhere else).
A further quote from the same section of the FHS suggests the use of /usr/var, which is a location we are not currently using in the Fedora Project, but seems like it would fit the requirements for both CoreOS and traditional RPM Fedora. This same hierarchy could be used for /var/log and /var/lib/libvirt/images as well.
The FHS reasoning ties back to the read-only /usr goal and sharing /usr among multiple systems, so you wouldn't want host-specifics written in there. I see that still applying to image-based management of /usr, honestly.
On Mon, Jan 10 2022 at 11:19:40 AM -0500, David Cantrell dcantrell@redhat.com wrote:
I would like to see Fedora introduce a new top-level directory called:
/state
Huh, a new top-level directory is a pretty big hammer. Seems like it would be a lot easier to just have two different locations for the rpmdb if non-ostree users really do not want it under /usr?
Michael
On Mon, Jan 10, 2022 at 5:20 PM David Cantrell dcantrell@redhat.com wrote:
On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
== Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE. [snip]
Moving the RPM database to /usr feels incorrect to me, but we should move it to gain the improvements as noted in the feature proposal.
Numerous followups have noted the requirement that /usr contain read-only content, that it be shareable across hosts, and similar concepts. While this may or may not doable now like we could in the past, the bigger thing to me is around the understanding of what /usr contains. It is generally understood that /usr contains [most of] the installed system. What I think is a bigger requirement or expection now is that one can tar up /usr and transport it to another system or virtual machine or container and expect that it will _probably_ work maybe with a bit of tinkering. This is a really valuable thing to have for developers. Moving the RPM database to this tree adds a component that is unnecessary and sort of out of place.
"OK, but you can do tar -X"
The tar example was simply that, an example. I feel the categorization of system data is more important here. Panu brought this up on the referenced RPM mailing list thread years ago. The original /var location for the RPM database was fine, but we're at a point where we kind of have multiple categories of things historically found in /var.
Reading comments and talking to people, the long standing understanding of /var is still "that's stuff you can rm -rf and the system will still work fine".
That's one interesting definition of "fine" if it's OK with wiping the system's (arguably) *only* non-disposable directory.
Technically you could remove the RPM database and the system still function, but arguably would still be broken since you really want the RPM database. This use case of removing the RPM database and still having a functioning system is really only useful for data recovery scenarios. You're ultimately going to reinstall. Probably.
So maybe the question is more "what is the correct location for data like the RPM database?" First, how is that data different from the rest of /var? It is host-specific, it's updated by tools we run all the time, and it's stateful. Losing it would render the system not really useful, though the system would still function. Am I missing anything here?
As Lennart noted, we have lots of examples in /usr of changing data. But I'd say the difference between those examples (library symlinks, cache files, etc) and the RPM database is that the loss of something like a library symlink or cache file can be repaired easily but if you lose the RPM database, the system is in an unrepairable state.
As another example... If you use Bitlbee, this would be like losing your account XML file in /var/lib/bitlbee. Sure, bitlbee still works, but that file is important for Bitlbee to work for you. You have to remember to hang on to that if you wipe /var or reinstall or whatever. I'd say the Bitlbee files in /var/lib/bitlbee also fit this slightly different /var data definition, just as an example.
"But what about the FHS?"
Ah, yes, the FHS. So, I am a fan of the FHS. I actually don't care that it doesn't change every week and is relatively stable. Nothing in the FHS prevents the addition of new top level directories.
I would prefer we steer this conversation in the direction of determining a new top level location to store data that fits this category of "stateful but variable".
/srv was introduced to provide a consistent location for data in this category for server daemons (except mail).
/media was introduced to provide a consistent location for removable media mount points since distributions all did things slightly differently.
/run was introduced for what was traditionally in /var/run.
"So what are you suggesting?"
I would like to see Fedora introduce a new top-level directory called:
/stateThat holds the RPM database and other variable and stateful data. This keeps it out of the /usr tree and can serve as a location for future data in this category.
If I ever see a system with /state I'll automatically assume this is the only location that's not wiped on a reboot, invented because the admin gave up on denylisting useless stuff in /var.
The whole proposal is kinda sad to read. It's 2022 and we're catering to filesystem-level rollbacks. Filesystem rollbacks *are* a gigantic unsubtle hammer that should not be used in an automated manner, or better yet, not used at all. As much as you don't do filesystem rollbacks to undo paragraph deletions in Libreoffice Writer, you shouldn't do filesystem rollbacks to fix your system. That's your package manager's / configuration manager's job.
Am 10.01.2022 um 18:57 schrieb Alexander Sosedkin asosedkin@redhat.com:
On Mon, Jan 10, 2022 at 5:20 PM David Cantrell dcantrell@redhat.com wrote:
On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
== Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE. [snip]
...
The whole proposal is kinda sad to read. It's 2022 and we're catering to filesystem-level rollbacks. Filesystem rollbacks *are* a gigantic unsubtle hammer that should not be used in an automated manner, or better yet, not used at all. As much as you don't do filesystem rollbacks to undo paragraph deletions in Libreoffice Writer, you shouldn't do filesystem rollbacks to fix your system. That's your package manager's / configuration manager's job.
Well, os-tree is doing filesystem rollback / -rollforward by architecture. If you want to feed rpm into it, it may be tricky.
But this argues for just moving rpmdb for rpm-oss-tree, since os-tree has its own logic anyway. No need and no advantage to do it for non-rpm-os-tree systems as well.
On Mon, Jan 10, 2022, at 11:19 AM, David Cantrell wrote:
On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
== Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE. [snip]
Moving the RPM database to /usr feels incorrect to me, but we should move it to gain the improvements as noted in the feature proposal.
Numerous followups have noted the requirement that /usr contain read-only content, that it be shareable across hosts, and similar concepts. While this may or may not doable now like we could in the past, the bigger thing to me is around the understanding of what /usr contains. It is generally understood that /usr contains [most of] the installed system. What I think is a bigger requirement or expection now is that one can tar up /usr and transport it to another system or virtual machine or container and expect that it will _probably_ work maybe with a bit of tinkering. This is a really valuable thing to have for developers.
No, this is not about developers tar'ing up `/usr` by hand. It's about cleanly separating who owns what, and what its lifecycle is, which is critcially important for both "image based" updates as well as "local client side snapshots". Both these approaches end up creating more than one copy of certain files.
See - https://blog.verbum.org/2020/08/22/immutable-%E2%86%92-reprovisionable-anti-... - http://lists.rpm.org/pipermail/rpm-maint/2021-December/018770.html
Moving the RPM database to this tree adds a component that is unnecessary and sort of out of place.
I struggle to how to even respond to this. Multiple independent groups who *actually work* on image based updates and/or client side snapshots all generally agree that the rpmdb should be in /usr.
That's where this Change came in.
On what basis are you making this claim "unnecessary and sort of out of place"?
"OK, but you can do tar -X"
The tar example was simply that, an example. I feel the categorization of system data is more important here. Panu brought this up on the referenced RPM mailing list thread years ago. The original /var location for the RPM database was fine, but we're at a point where we kind of have multiple categories of things historically found in /var.
Reading comments and talking to people, the long standing understanding of /var is still "that's stuff you can rm -rf and the system will still work fine".
Yes, we call this "factory reset". https://github.com/coreos/fedora-coreos-tracker/issues/399 I am not sure where the terminology came from, but it is widely used when talking about e.g. mobile phones today.
Technically you could remove the RPM database and the system still function, but arguably would still be broken since you really want the RPM database. This use case of removing the RPM database and still having a functioning system is really only useful for data recovery scenarios. You're ultimately going to reinstall. Probably.
At least from my PoV, the rpmdb is by default read-only (except to rpm-ostree) along with the rest of /usr, so you can't just rm -rf it alone.
So maybe the question is more "what is the correct location for data like the RPM database?" First, how is that data different from the rest of /var? It is host-specific,
No. An image based updates model (both ostree and container images) ships a pristine copy that is bit-for-bit the same on every system. Container runtimes tend to make it mutable by default of course, but that doesn't change the fact that it's not by default host specific.
I would like to see Fedora introduce a new top-level directory called:
/state
We have years of investment in rpm-ostree in the current design. For example, the tooling supports `rpm-ostree db diff` which shows you the delta between the current and pending rpmdb. How would this new directory work? How would it better solve problems that are today solved with the location in /usr? And do you even have a sense of how much work would creating for the rpm-ostree stack at least with a new toplevel directory with new, as yet ill-defined semantics?
On Mon, Jan 10, 2022 at 01:24:20PM -0500, Colin Walters wrote:
On Mon, Jan 10, 2022, at 11:19 AM, David Cantrell wrote:
On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
== Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE. [snip]
Moving the RPM database to /usr feels incorrect to me, but we should move it to gain the improvements as noted in the feature proposal.
Numerous followups have noted the requirement that /usr contain read-only content, that it be shareable across hosts, and similar concepts. While this may or may not doable now like we could in the past, the bigger thing to me is around the understanding of what /usr contains. It is generally understood that /usr contains [most of] the installed system. What I think is a bigger requirement or expection now is that one can tar up /usr and transport it to another system or virtual machine or container and expect that it will _probably_ work maybe with a bit of tinkering. This is a really valuable thing to have for developers.
No, this is not about developers tar'ing up `/usr` by hand. It's about cleanly separating who owns what, and what its lifecycle is, which is critcially important for both "image based" updates as well as "local client side snapshots". Both these approaches end up creating more than one copy of certain files.
I was trying to give an additional example of what people may do with /usr and the expectations around it. /usr contains data that comes from the distribution. We drop packages in place and we get files from that. The RPM database is generated and updated on the target system and represents the actions of RPM on that system. This is why I categorize the RPM database differently than the rest of what is in /usr.
See
- https://blog.verbum.org/2020/08/22/immutable-%E2%86%92-reprovisionable-anti-...
- http://lists.rpm.org/pipermail/rpm-maint/2021-December/018770.html
Moving the RPM database to this tree adds a component that is unnecessary and sort of out of place.
I struggle to how to even respond to this. Multiple independent groups who *actually work* on image based updates and/or client side snapshots all generally agree that the rpmdb should be in /usr.
That's where this Change came in.
On what basis are you making this claim "unnecessary and sort of out of place"?
See above. The RPM database contains data generated by RPM on the system in question, it's not delivered to the user in a package or some other form.
The change proposal is about moving the RPM database to /usr. I am offering my comments on why I think /usr is the incorrect location for this data.
"OK, but you can do tar -X"
The tar example was simply that, an example. I feel the categorization of system data is more important here. Panu brought this up on the referenced RPM mailing list thread years ago. The original /var location for the RPM database was fine, but we're at a point where we kind of have multiple categories of things historically found in /var.
Reading comments and talking to people, the long standing understanding of /var is still "that's stuff you can rm -rf and the system will still work fine".
Yes, we call this "factory reset". https://github.com/coreos/fedora-coreos-tracker/issues/399 I am not sure where the terminology came from, but it is widely used when talking about e.g. mobile phones today.
Technically you could remove the RPM database and the system still function, but arguably would still be broken since you really want the RPM database. This use case of removing the RPM database and still having a functioning system is really only useful for data recovery scenarios. You're ultimately going to reinstall. Probably.
At least from my PoV, the rpmdb is by default read-only (except to rpm-ostree) along with the rest of /usr, so you can't just rm -rf it alone.
Except removing a package would write that change to the rpmdb.
So maybe the question is more "what is the correct location for data like the RPM database?" First, how is that data different from the rest of /var? It is host-specific,
No. An image based updates model (both ostree and container images) ships a pristine copy that is bit-for-bit the same on every system. Container runtimes tend to make it mutable by default of course, but that doesn't change the fact that it's not by default host specific.
We have other Fedora editions besides rpm-ostree. This change affects everyone.
I would like to see Fedora introduce a new top-level directory called:
/stateWe have years of investment in rpm-ostree in the current design. For example, the tooling supports `rpm-ostree db diff` which shows you the delta between the current and pending rpmdb. How would this new directory work? How would it better solve problems that are today solved with the location in /usr? And do you even have a sense of how much work would creating for the rpm-ostree stack at least with a new toplevel directory with new, as yet ill-defined semantics?
The proposal has asked to move the RPM database. I don't think /usr is the correct location. But I agree that /var could use some reorganization. I feel we have an opportunity to define a set of data that represents the system in question and define a more correct location from it while not redefining /usr or /var.
Thanks,
On Thu, Jan 13, 2022 at 8:34 AM David Cantrell dcantrell@redhat.com wrote:
On Mon, Jan 10, 2022 at 01:24:20PM -0500, Colin Walters wrote:
No, this is not about developers tar'ing up `/usr` by hand. It's about cleanly separating who owns what, and what its lifecycle is, which is critcially important for both "image based" updates as well as "local client side snapshots". Both these approaches end up creating more than one copy of certain files.
I was trying to give an additional example of what people may do with /usr and the expectations around it. /usr contains data that comes from the distribution. We drop packages in place and we get files from that. The RPM database is generated and updated on the target system and represents the actions of RPM on that system. This is why I categorize the RPM database differently than the rest of what is in /usr.
Does rpm-ostree has the rpmdb in the wrong place? Or is the OSTree model OK because of where rpmdb was created, or how it's delivered, not because of the intrinsic nature of the rpmdb file?
On what basis are you making this claim "unnecessary and sort of out of place"?
See above. The RPM database contains data generated by RPM on the system in question, it's not delivered to the user in a package or some other form.
It is delivered to the user in some other form on rpm-ostree systems, not generated on the system in question. Does the location or delivery mechanism of the rpmdb alter the appropriateness of /usr as a location? Why does rpmdb's origin act as a material fact in determining where it should be stored rather than its intrinsic nature?
I do not think the FHS helps us resolve where rpmdb goes because the FHS's own history of changes shows how often it made the wrong call. It follows reality rather than leading it. By nature it describes a universe. One system. Not many systems, not a multiverse.
FHS: "/usr/lib includes object files and libraries. On some systems, it may also include internal binaries that are not intended to be executed directly by users or shell scripts."
It is in effect anything that has to do with system function. There's a boat load of man pages, images, source files, in /usr. Clearly I can add to /usr outside of RPM by compiling and installing the kernel from upstream git. Binaries subsequently appear in /usr as a result of that action, but not in rpmdb. Therefore parity between the binaries in /usr and what's reflected in rpmdb is not an argument for whether a thing is appropriate for /usr or not. Nor where rpmdb is built or how it's delivered.
5.8. /var/lib : Variable state information 5.8.1. Purpose "State information is generally used to preserve the condition of an application (or a group of inter-related applications) between invocations and between different instances of the same application. State information should generally remain valid after a reboot, should not be logging output, and should not be spooled data."
The rpmdb is most often *invariant*. Or more correctly, its variance is directly associated with some other variance, like a package being added, modified, or removed. It doesn't get modified as a matter of course like a VM image, or http database, or logs.
Since the FHS provides no guidance on /state, I have to agree that /state is a better location than /var on the face of it. Chances are /var was never really a very good location for rpmdb in the first place, but it only becomes obvious upon system snapshots creating a multiverse of systems. There isn't a single system on a system with system snapshots. There are now multiple systems of different revisions. And that suggests multiple rpmdb's of different revisions to describe them. Or perhaps a single rpmdb that is snapshot aware, containing information about all system snapshots.
So I think we are better off trying to understand the relative consequences of /state versus /usr for rpmdb than appealing to any higher authority (like the FHS), or feelings. I can't act on either the FHS or feelings, they're both unreliable.
At least from my PoV, the rpmdb is by default read-only (except to rpm-ostree) along with the rest of /usr, so you can't just rm -rf it alone.
Except removing a package would write that change to the rpmdb.
Removing a package that contains files in /usr involves writes to /usr regardless of where rpmdb is located, because any modification of /usr means /usr must be read-write. Add, modify or remove.
Even our default mount option, relatime, results in writes to /usr every day just by using the system.
We have years of investment in rpm-ostree in the current design. For example, the tooling supports `rpm-ostree db diff` which shows you the delta between the current and pending rpmdb. How would this new directory work? How would it better solve problems that are today solved with the location in /usr? And do you even have a sense of how much work would creating for the rpm-ostree stack at least with a new toplevel directory with new, as yet ill-defined semantics?
The proposal has asked to move the RPM database. I don't think /usr is the correct location.
I understand the words. I still don't follow the logic of why it's not correct, in particular if it is correct for rpm-ostree based systems but not correct for (traditional) RPM based systems.
A sister distribution has made this decision already and not in a vacuum. All the same information available to us is available to them. Yet they are saying /usr/lib/sysimage is the correct location for rpmdb. It requires a stronger argument than provided so far to convince me they arrived at the wrong conclusion.
-- Chris Murphy
On Mon, Jan 10, 2022 at 9:20 AM David Cantrell dcantrell@redhat.com wrote:
On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
== Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE. [snip]
Moving the RPM database to /usr feels incorrect to me, but we should move it to gain the improvements as noted in the feature proposal.
Going back to the original discussions on moving rpmdb...
Preferred is a new top-level location in /usr, .e.g /usr/sysimage/rpm. Next is "least worst" in /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006764.html http://lists.rpm.org/pipermail/rpm-maint/2017-October/006722.html
And the convergence was on /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006785.html
I don't see how /state solves the problem, rather than just rearranging the chairs.
Numerous followups have noted the requirement that /usr contain read-only content, that it be shareable across hosts, and similar concepts. While this may or may not doable now like we could in the past, the bigger thing to me is around the understanding of what /usr contains. It is generally understood that /usr contains [most of] the installed system. What I think is a bigger requirement or expection now is that one can tar up /usr and transport it to another system or virtual machine or container and expect that it will _probably_ work maybe with a bit of tinkering. This is a really valuable thing to have for developers. Moving the RPM database to this tree adds a component that is unnecessary and sort of out of place.
Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt?
If /usr is to be truly portable and have e.g. 'rpm query, verify, remove, reinstall' work as expected, you need the metadata (the database) representing its state to always come along for the ride. Either the database is already in /usr, or you have to make sure /usr and /state are inseparable.
If /usr and /state are inseparable, and if rpm can also describe anything in /etc or /var or /opt, then all or part of those directories are also inseparable from /state. And thus /usr. So I think /state doesn't help.
To what degree do rpm and dnf intend to touch locations outside of /usr *and care* about tracking those changes? I think rpm can't remain static for all time. It either needs to become aware of multiple root trees, and even mix and match top-level directories to create variable roots. And possibly even manage these things. Or it needs to constrain its reach to /usr and /opt. Whether /usr and /opt are tied together, or can be mix and match with their own rpmdb's, I have no strong opinion on.
On 1/10/22 23:53, Chris Murphy wrote:
On Mon, Jan 10, 2022 at 9:20 AM David Cantrell dcantrell@redhat.com wrote:
On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
== Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE. [snip]
Moving the RPM database to /usr feels incorrect to me, but we should move it to gain the improvements as noted in the feature proposal.
Going back to the original discussions on moving rpmdb...
Preferred is a new top-level location in /usr, .e.g /usr/sysimage/rpm. Next is "least worst" in /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006764.html http://lists.rpm.org/pipermail/rpm-maint/2017-October/006722.html
And the convergence was on /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006785.html
I don't see how /state solves the problem, rather than just rearranging the chairs.
The problem with /usr/something is that the rpmdb is not specific to /usr contents at all, and unlike any other content in there, so putting it there just *feels so wrong*. That's what /state or /sysimage or, as we now have, /var supposedly solves.
I thought I'd suggested something at / level back then (in https://lists.rpm.org/pipermail/rpm-maint/2017-October/006697.html and/or https://lists.rpm.org/pipermail/rpm-maint/2017-October/006699.html) but seems like memory is failing me :) Maybe I thought it would seem too outrageous to FHS believers to bother.
The point was though, that the rpmdb is not at all the only data of this kind and so having a dedicated home makes sense.
For many practical purposes it's probably just rearranging the chairs, but a separate top-level directory describing the *system* state seems instinctively *much* more correct solution to it than stuffing it somewhere deep inside a loosely related fs.
Just FWIW, I would quit my whining about this right there if it went to a new toplevel directory instead because it just *feels* right unlike /usr.
Numerous followups have noted the requirement that /usr contain read-only content, that it be shareable across hosts, and similar concepts. While this may or may not doable now like we could in the past, the bigger thing to me is around the understanding of what /usr contains. It is generally understood that /usr contains [most of] the installed system. What I think is a bigger requirement or expection now is that one can tar up /usr and transport it to another system or virtual machine or container and expect that it will _probably_ work maybe with a bit of tinkering. This is a really valuable thing to have for developers. Moving the RPM database to this tree adds a component that is unnecessary and sort of out of place.
Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt?
If /usr is to be truly portable and have e.g. 'rpm query, verify, remove, reinstall' work as expected, you need the metadata (the database) representing its state to always come along for the ride. Either the database is already in /usr, or you have to make sure /usr and /state are inseparable.
If /usr and /state are inseparable, and if rpm can also describe anything in /etc or /var or /opt, then all or part of those directories are also inseparable from /state. And thus /usr. So I think /state doesn't help.
For one, /state (or whatever toplevel directory) allows for the fact that there are write-operations to rpmdb that do not touch any external files while maintaining read-only /usr. Such as rpmdb --rebuilddb, or rpm --import.
And like mentioned in the original discussion and now here, although the discussion is on rpmdb, it's not the only data of this kind.
To what degree do rpm and dnf intend to touch locations outside of /usr *and care* about tracking those changes?
I don't understand the question. Rpm tracks and cares about all content it knows about equally, regardless of the path. /usr is NOT special in any way to rpm, it's just that most of *distro* content ends up in there but a huge number of packages have content spread across /etc too.
I think rpm can't remain static for all time. It either needs to become aware of multiple root trees, and even mix and match top-level directories to create variable roots. And possibly even manage these things. Or it needs to constrain its reach to /usr and /opt. Whether /usr and /opt are tied together, or can be mix and match with their own rpmdb's, I have no strong opinion on.
Oh, multiple rpmdbs. It's a while since that last turned up. It gets tossed around as a solution but to me it looks like it brings more problems than it solves.
- Panu -
On Tue, Jan 11, 2022, at 4:00 AM, Panu Matilainen wrote:
The point was though, that the rpmdb is not at all the only data of this kind and so having a dedicated home makes sense.
You mentioned dnf/yum/PackageKit data; there's two kinds of that. One is e.g. /var/cache/dnf which does *not* move. It's just a cache. (Now there is this whole other very interesting thread around "why don't we include a cache of the rpm-md inside e.g. ostree or container images? And the reason why not is we don't want to have to respin all images every time a package not in the image changes)
dnf does have its own outside-of-rpm state database in /var/lib/dnf which is closer to this. The situation with that is messier. AIUI this proposal so far is not calling for moving this. Where it lives and how it's versioned has strong implications for the people who want to support snapshot/rollback. But it's not relevant for rpm-ostree, which does not use this part of libdnf. We maintain our own little "state file" - for lots more detail on this, see https://github.com/coreos/rpm-ostree/issues/2326 (And it's important to note that rpm-ostree's origin file has almost nothing *unless* one explicitly engages client-side layering/overrides)
For one, /state (or whatever toplevel directory) allows for the fact that there are write-operations to rpmdb that do not touch any external files while maintaining read-only /usr. Such as rpmdb --rebuilddb,
`rpmdb --rebuilddb` is basically about supporting switching from e.g. bdb to sqlite, right? On the rpm-ostree side at least, the default format comes from the base image; there's no reason to directly support `rpmdb --rebuilddb` as something any normal user or admin would need.
or rpm --import.
OK yes, this (along with rpms that install into /opt as you mentioned) are I think the biggest examples of "rpmdb has stuff not about /usr".
rpm --import actually encapsulates really well all the problems and benefits with everything we're trying to do. I have a big related blurb here https://github.com/rpm-software-management/libdnf/issues/43
But as I understand it, the creation of /etc/pki/rpm-gpg was at least partially motivated by the fact that in the traditional RPM model, the fact that GPG keys are stored in the rpmdb meant there was no way to update them "in band" of a default rpm operation. Today Fedora ships these GPG keys as an RPM which hence contain files, and when you type `yum upgrade` (or rpm-ostree upgrade) you get updates to those files, the same as other files. Now, as noted in the issue PackageKit (whose code was the precursor to libdnf, which has the code that rpm-ostree uses but AFAIK still not current dnf which has this logic in Python) auto-imports all of them. I am not completely sure how updating the rpmdb with new e.g. Fedora GPG key works. It might be part of system-upgrade or something?
And then this all relates to a higher level goal we have with "image based updates", which is avoiding (or minimizing as much as possible) per-system hysteresis or "unmanaged state", particularly opaque (hard to see/diagnose/inspect) state.
This set of trusted GPG keys stored in the rpmdb is both. The set of keys will just keep growing across in-place upgrades, depending on the initial Fedora version installed. And wh And this is security-relevant state, it's the set of trusted keys for RPM. Now, I am sure a number of sysadmins do understand that the rpmdb contains GPG keys. I'd bet a whole lot don't. I definitely think that it's confusing to have both /etc/pki/rpm-gpg *and* keys stored in the rpmdb. Of the two, I think the former - i.e. text files one can edit with standard tools - is much easier to understand and work with. It's also already in a place that is designed for users to edit in `/etc`.
So by moving the rpmdb to /usr, it's basically saying that `rpm --import` should change.
That said, this design could also clearly use some "systemd style" config model. ostree already supports /usr/share/ostree/trusted.gpg.d which is designed for keys shipped by the OS vendor. But, what's really tricky about this is we want to support in-place updates from previous versions (i.e. at least N-1), but hopefully not too old versions. Well, this is leading to a tangent so I'll cut off this sub-thread.
The TL;DR for me is: I think everyone agrees that moving the rpmdb as it is today to /usr is not 100% a perfect fit. But it's a ~90% fit - almost all the raw data is just headers which are clearly immutable data generated elsewhere. And by making this change, we're basically saying we'll fix the remaining 10% of cases.
Note for the people who are trying to do "client side snapshot/rollback" (i.e. the people whose names are attached to the Change), the rpmdb is still mutable by default. And I think their idea is is that by doing a "snapshot, then upgrade in place via traditional rpm/yum methods" approach, the blast radius is then contained to what happens on explicit rollback. And that's then very limited to e.g. "I wanted to add a 3rd party RPM and did `rpm --import` to trust their key and then rolled back and lost that change" which is clearly not a big deal.
For rpm-ostree's "image based" updates model, to avoid hysteresis we always construct a *new* root (just hardlinked) so any manual outside-of-rpm-ostree's control changes to `/usr` will always be thrown away, and that's why it's read-only. So having `rpm --import` fail is intentional today.
But, I hope we can get agreement about something like having `rpm --import` write to `/etc/pki/rpm-gpg` and dropping gpg keys from the rpmdb.
On 1/11/22 17:18, Colin Walters wrote:
On Tue, Jan 11, 2022, at 4:00 AM, Panu Matilainen wrote:
The point was though, that the rpmdb is not at all the only data of this kind and so having a dedicated home makes sense.
You mentioned dnf/yum/PackageKit data; there's two kinds of that. One is e.g. /var/cache/dnf which does *not* move. It's just a cache. (Now there is this whole other very interesting thread around "why don't we include a cache of the rpm-md inside e.g. ostree or container images? And the reason why not is we don't want to have to respin all images every time a package not in the image changes)
dnf does have its own outside-of-rpm state database in /var/lib/dnf which is closer to this. The situation with that is messier. AIUI this proposal so far is not calling for moving this. Where it lives and how it's versioned has strong implications for the people who want to support snapshot/rollback. But it's not relevant for rpm-ostree, which does not use this part of libdnf. We maintain our own little "state file" - for lots more detail on this, see https://github.com/coreos/rpm-ostree/issues/2326 (And it's important to note that rpm-ostree's origin file has almost nothing *unless* one explicitly engages client-side layering/overrides)
For one, /state (or whatever toplevel directory) allows for the fact that there are write-operations to rpmdb that do not touch any external files while maintaining read-only /usr. Such as rpmdb --rebuilddb,
`rpmdb --rebuilddb` is basically about supporting switching from e.g. bdb to sqlite, right? On the rpm-ostree side at least, the default format comes from the base image; there's no reason to directly support `rpmdb --rebuilddb` as something any normal user or admin would need.
No, it's a database maintenance/repair operation. Converting between formats is merely a convenient side-effect.
or rpm --import.
OK yes, this (along with rpms that install into /opt as you mentioned) are I think the biggest examples of "rpmdb has stuff not about /usr".
rpm --import actually encapsulates really well all the problems and benefits with everything we're trying to do. I have a big related blurb here https://github.com/rpm-software-management/libdnf/issues/43
But as I understand it, the creation of /etc/pki/rpm-gpg was at least partially motivated by the fact that in the traditional RPM model, the fact that GPG keys are stored in the rpmdb meant there was no way to update them "in band" of a default rpm operation. Today Fedora ships these GPG keys as an RPM which hence contain files, and when you type `yum upgrade` (or rpm-ostree upgrade) you get updates to those files, the same as other files. Now, as noted in the issue PackageKit (whose code was the precursor to libdnf, which has the code that rpm-ostree uses but AFAIK still not current dnf which has this logic in Python) auto-imports all of them. I am not completely sure how updating the rpmdb with new e.g. Fedora GPG key works. It might be part of system-upgrade or something?
And then this all relates to a higher level goal we have with "image based updates", which is avoiding (or minimizing as much as possible) per-system hysteresis or "unmanaged state", particularly opaque (hard to see/diagnose/inspect) state.
This set of trusted GPG keys stored in the rpmdb is both. The set of keys will just keep growing across in-place upgrades, depending on the initial Fedora version installed. And wh And this is security-relevant state, it's the set of trusted keys for RPM. Now, I am sure a number of sysadmins do understand that the rpmdb contains GPG keys. I'd bet a whole lot don't. I definitely think that it's confusing to have both /etc/pki/rpm-gpg *and* keys stored in the rpmdb. Of the two, I think the former - i.e. text files one can edit with standard tools - is much easier to understand and work with. It's also already in a place that is designed for users to edit in `/etc`.
So by moving the rpmdb to /usr, it's basically saying that `rpm --import` should change.
This doesn't seem to be documented as a dependency of this move...
That said, this design could also clearly use some "systemd style" config model. ostree already supports /usr/share/ostree/trusted.gpg.d which is designed for keys shipped by the OS vendor. But, what's really tricky about this is we want to support in-place updates from previous versions (i.e. at least N-1), but hopefully not too old versions. Well, this is leading to a tangent so I'll cut off this sub-thread.
The TL;DR for me is: I think everyone agrees that moving the rpmdb as it is today to /usr is not 100% a perfect fit. But it's a ~90% fit - almost all the raw data is just headers which are clearly immutable data generated elsewhere. And by making this change, we're basically saying we'll fix the remaining 10% of cases.
Note for the people who are trying to do "client side snapshot/rollback" (i.e. the people whose names are attached to the Change), the rpmdb is still mutable by default. And I think their idea is is that by doing a "snapshot, then upgrade in place via traditional rpm/yum methods" approach, the blast radius is then contained to what happens on explicit rollback. And that's then very limited to e.g. "I wanted to add a 3rd party RPM and did `rpm --import` to trust their key and then rolled back and lost that change" which is clearly not a big deal.
For rpm-ostree's "image based" updates model, to avoid hysteresis we always construct a *new* root (just hardlinked) so any manual outside-of-rpm-ostree's control changes to `/usr` will always be thrown away, and that's why it's read-only. So having `rpm --import` fail is intentional today.
But, I hope we can get agreement about something like having `rpm --import` write to `/etc/pki/rpm-gpg` and dropping gpg keys from the rpmdb.
Rpm actually supports a keyring from the fs since 4.6 I think, it's just not integrated with --import and doesn't play that well along with the rest of the ecosystem because everything expects those gpg-pubkey "packages". We had bugs filed because somebody tried to use the fs-based thing but yum broke up.
The problem with fs based keyring is that packages can drop arbitrary unrelated keys in there, combined with the rpm model of imported == trusted. The trust aspect would have to be maintained in a place not directly accessible to packages.
That imported == trusted implication is of course a long-standing flaw in rpm but difficult to change because of all the assumptions that have been built around it. Everybody (me included) loves to hate the gpg-pubkey headers as we know them, but rpmdb as a keyring storage does make sense. I mean, rpmdb is data which rpm knows, and is free to store whatever it wishes there. Do understand that Fedora or anybody else moving it to /usr will not prevent rpm from adding more features that may need to modify it to store further data. So while it may seem "okay" now, it may as well not be that way in the future.
- Panu -
- Panu -
On Wed, Jan 12, 2022 at 1:36 AM Panu Matilainen pmatilai@redhat.com wrote:
So by moving the rpmdb to /usr, it's basically saying that `rpm --import` should change.
This doesn't seem to be documented as a dependency of this move...
Added. https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Dependencies
On Wed, Jan 12, 2022 at 3:54 AM Chris Murphy lists@colorremedies.com wrote:
On Wed, Jan 12, 2022 at 1:36 AM Panu Matilainen pmatilai@redhat.com wrote:
So by moving the rpmdb to /usr, it's basically saying that `rpm --import` should change.
This doesn't seem to be documented as a dependency of this move...
Added. https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Dependencies
No. This is nonsense. In current Fedora, this doesn't matter. In a hypothetical future model of a transactional system, we'd want to move repo configs and gpg key files from /etc to /usr at the same time.
Right now, this is out of scope because the DNF team is not ready for that work.
On Wed, Jan 12, 2022 at 2:57 PM Neal Gompa ngompa13@gmail.com wrote:
On Wed, Jan 12, 2022 at 3:54 AM Chris Murphy lists@colorremedies.com wrote:
On Wed, Jan 12, 2022 at 1:36 AM Panu Matilainen pmatilai@redhat.com wrote:
So by moving the rpmdb to /usr, it's basically saying that `rpm --import` should change.
This doesn't seem to be documented as a dependency of this move...
Added. https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Dependencies
No. This is nonsense. In current Fedora, this doesn't matter. In a hypothetical future model of a transactional system, we'd want to move repo configs and gpg key files from /etc to /usr at the same time.
Right now, this is out of scope because the DNF team is not ready for that work.
Ahh yeah fair enough.
There are further discussions before we actually get to snapshot and rollbacks, not least of which is things like 'dnf history' can't show you an accurate history if there's been an external (filesystem) rollback of the system root that does not rollback /var/lib/dnf. Ideally we'd never have to rollback anything in /var, but possibly with an option for selective restore.
On Tue, Jan 11, 2022 at 2:00 AM Panu Matilainen pmatilai@redhat.com wrote:
The problem with /usr/something is that the rpmdb is not specific to /usr contents at all, and unlike any other content in there, so putting it there just *feels so wrong*. That's what /state or /sysimage or, as we now have, /var supposedly solves.
I thought I'd suggested something at / level back then (in https://lists.rpm.org/pipermail/rpm-maint/2017-October/006697.html and/or https://lists.rpm.org/pipermail/rpm-maint/2017-October/006699.html) but seems like memory is failing me :) Maybe I thought it would seem too outrageous to FHS believers to bother.
The point was though, that the rpmdb is not at all the only data of this kind and so having a dedicated home makes sense.
For many practical purposes it's probably just rearranging the chairs, but a separate top-level directory describing the *system* state seems instinctively *much* more correct solution to it than stuffing it somewhere deep inside a loosely related fs.
If rpm is constrained to /usr, then its database can properly be somewhere in /usr
If rpm is unconstrained, with transactions touching any of /usr /var /opt /etc, the resulting paradigm is inherently complicated, and an additional top-level directory doesn't help make it less complicated.
Either top-level directories have their own databases, so rpm knows what's in them even if one is rolled back but not another; or there's one database to describe all of these top-level directories which then need to share a single (sub)volume so they're all snapshot and rolled back together. Either way, there's additionally a need for carve outs for things that shouldn't be rolled back.
We have an example of the latter. (open)SUSE's current layout. There's a 10 line /etc/fstab, 9 of which are various Btrfs subvolumes to achieve the necessary carve out. It's sufficiently complex that the direction they're looking to go in is one in which rpm's only touch /usr. There's /usr/etc for read-only systemd defaults. And local customizations go in /etc. Much like rpm-ostree.
Projects within two rpm-based distros independently came to realize that unconstrained rpm is difficult. Where rpm-ostree decided to do the hard work of actually becoming aware of the reality there's multiple system roots.
I don't understand the question. Rpm tracks and cares about all content it knows about equally, regardless of the path. /usr is NOT special in any way to rpm, it's just that most of *distro* content ends up in there but a huge number of packages have content spread across /etc too.
I think rpm can't remain static for all time. It either needs to become aware of multiple root trees, and even mix and match top-level directories to create variable roots. And possibly even manage these things. Or it needs to constrain its reach to /usr and /opt. Whether /usr and /opt are tied together, or can be mix and match with their own rpmdb's, I have no strong opinion on.
Oh, multiple rpmdbs. It's a while since that last turned up. It gets tossed around as a solution but to me it looks like it brings more problems than it solves.
Given the choice, I prefer rpm only touches /usr, which includes /usr/var and /usr/etc.
But if left unconstrained, having databases inside the directories they describe is more reliable than treating the database as a sidecar file that can much more easily become disconnected with one or more top-level directories it ostensibly describes the contents of.
On 1/12/22 10:45, Chris Murphy wrote:
On Tue, Jan 11, 2022 at 2:00 AM Panu Matilainen pmatilai@redhat.com wrote:
The problem with /usr/something is that the rpmdb is not specific to /usr contents at all, and unlike any other content in there, so putting it there just *feels so wrong*. That's what /state or /sysimage or, as we now have, /var supposedly solves.
I thought I'd suggested something at / level back then (in https://lists.rpm.org/pipermail/rpm-maint/2017-October/006697.html and/or https://lists.rpm.org/pipermail/rpm-maint/2017-October/006699.html) but seems like memory is failing me :) Maybe I thought it would seem too outrageous to FHS believers to bother.
The point was though, that the rpmdb is not at all the only data of this kind and so having a dedicated home makes sense.
For many practical purposes it's probably just rearranging the chairs, but a separate top-level directory describing the *system* state seems instinctively *much* more correct solution to it than stuffing it somewhere deep inside a loosely related fs.
If rpm is constrained to /usr, then its database can properly be somewhere in /usr
If rpm is unconstrained, with transactions touching any of /usr /var /opt /etc, the resulting paradigm is inherently complicated, and an additional top-level directory doesn't help make it less complicated.
WTF is this talk about constraining? There's no such constraint in rpm, and never will. And Fedora packages are not constrained there either, a vast percentage places stuff in /etc, touches directory tree in /var and then there's /boot and the software collections and third party software.
Either top-level directories have their own databases, so rpm knows what's in them even if one is rolled back but not another; or there's one database to describe all of these top-level directories which then need to share a single (sub)volume so they're all snapshot and rolled back together. Either way, there's additionally a need for carve outs for things that shouldn't be rolled back.
Erm. Packages don't live conveniently inside a single top level tree, they can't be split up that way. Many if not most span at least /etc and /usr, and /usr may be further split into separate mounts (eg /usr/local or /usr/share). And many things which install to /opt also place things into /usr for desktop integration.
We have an example of the latter. (open)SUSE's current layout. There's a 10 line /etc/fstab, 9 of which are various Btrfs subvolumes to achieve the necessary carve out. It's sufficiently complex that the direction they're looking to go in is one in which rpm's only touch /usr. There's /usr/etc for read-only systemd defaults. And local customizations go in /etc. Much like rpm-ostree.
Projects within two rpm-based distros independently came to realize that unconstrained rpm is difficult. Where rpm-ostree decided to do the hard work of actually becoming aware of the reality there's multiple system roots.
I don't understand the question. Rpm tracks and cares about all content it knows about equally, regardless of the path. /usr is NOT special in any way to rpm, it's just that most of *distro* content ends up in there but a huge number of packages have content spread across /etc too.
I think rpm can't remain static for all time. It either needs to become aware of multiple root trees, and even mix and match top-level directories to create variable roots. And possibly even manage these things. Or it needs to constrain its reach to /usr and /opt. Whether /usr and /opt are tied together, or can be mix and match with their own rpmdb's, I have no strong opinion on.
Oh, multiple rpmdbs. It's a while since that last turned up. It gets tossed around as a solution but to me it looks like it brings more problems than it solves.
Given the choice, I prefer rpm only touches /usr, which includes /usr/var and /usr/etc.
But if left unconstrained, having databases inside the directories they describe is more reliable than treating the database as a sidecar file that can much more easily become disconnected with one or more top-level directories it ostensibly describes the contents of.
Here seems to be another SMALL undocumented dependency of this change: completing the /usrmove thing to cover the whole world including /opt, /etc, /var, and presumably /boot as well because packages put stuff in it.
Rpm wont care if you want to move content from /etc to /usr/etc and ditto for /var, /opt and /boot which is also used by packages, ie to complete to complete the great /usrmove thing started a decade ago. But that's quite a hidden agenda in this change if you ask me.
- Panu -
On Wed, Jan 12, 2022, at 4:04 AM, Panu Matilainen wrote:
Here seems to be another SMALL undocumented dependency of this change: completing the /usrmove thing to cover the whole world including /opt, /etc, /var, and presumably /boot as well because packages put stuff in it.
There are very few packages that put things in /boot. For example, fwupd used to be in that set, but moved into a "self updater" model where the binary goes in /usr, and then it copies itself into the ESP instead of having yum/rpm do it.
Now, rpm-ostree also does this with /boot: https://github.com/coreos/rpm-ostree/blob/210bf148342a9545c9841ae6d8403354ce...
And then we have a sister project https://github.com/coreos/bootupd/ that is only shipped in FCOS today (but would make sense to use on everything that uses rpm-ostree) which is scoped explicitly to only handle stuff on the ESP (and eventually, stuff like grub on the MBR and other architectures too).
Correct handling of /boot is obviously essential for transactional updates; ostree is entirely designed around a "strong binding" of (kernel, userspace) pairs. Handling kernels in /boot for "client side snapshots" is something most projects in this space do out of band, as far as I've seen.
Again, it's really that 90% of the data in the rpmdb is for /usr. We recently changed the kernel RPM to stick the kernel binary in /usr/lib/modules/$kver and only *copy* it from there to strengthen this model. And this move is pushing things farther along in that direction.
On Wed, Jan 12, 2022 at 2:04 AM Panu Matilainen pmatilai@redhat.com wrote:
On 1/12/22 10:45, Chris Murphy wrote:
On Tue, Jan 11, 2022 at 2:00 AM Panu Matilainen pmatilai@redhat.com wrote:
For many practical purposes it's probably just rearranging the chairs, but a separate top-level directory describing the *system* state seems instinctively *much* more correct solution to it than stuffing it somewhere deep inside a loosely related fs.
If rpm is constrained to /usr, then its database can properly be somewhere in /usr
If rpm is unconstrained, with transactions touching any of /usr /var /opt /etc, the resulting paradigm is inherently complicated, and an additional top-level directory doesn't help make it less complicated.
WTF is this talk about constraining? There's no such constraint in rpm, and never will.
I'm not talking about changing the nature of the beast. I'm talking about how it's used as a tool in a distribution with a lot of other requirements and tradeoffs.
Expanding rpm is also a valid path. We have an example "DNF snapper plug-in" that demonstrates at least DNF *could* get into the business of managing snapshots and rollbacks, if it wants that responsibility, if it's helpful and worth the effort, and if the resources exist to do it. That's an interesting discussion.
And Fedora packages are not constrained there either, a vast percentage places stuff in /etc, touches directory tree in /var and then there's /boot and the software collections and third party software.
/boot might need its own manager(s). I'm strongly leaning toward bootupd and fwupd being the only things that touch /boot and everything else is proscribed.
/boot is a small but hot mess right now. RPM, grub, grubby, dracut, all can touch /boot and in the Boot Loader Spec context there's multiple OS's sharing /boot. About the only thing I like better is linux as the bootloader, e.g. Linux Boot, so we have real device and file system drivers. Anything that manages /boot needs to needs understand snapshots, and which kernel+initramfs can boot which snapshots.
Either top-level directories have their own databases, so rpm knows what's in them even if one is rolled back but not another; or there's one database to describe all of these top-level directories which then need to share a single (sub)volume so they're all snapshot and rolled back together. Either way, there's additionally a need for carve outs for things that shouldn't be rolled back.
Erm. Packages don't live conveniently inside a single top level tree, they can't be split up that way.
Right. I view this as a criticism, not a benefit though. Neither rpm nor the FHS are organizing things in a way that helps us do rollbacks. They're impediments we keep having to take other shortcuts and workarounds to achieve that goal, and as yet they still have too many shortcomings. It really isn't to be underestimated the importance of multiple independent parties coming to the same kinds of conclusions.
Given the choice, I prefer rpm only touches /usr, which includes /usr/var and /usr/etc.
But if left unconstrained, having databases inside the directories they describe is more reliable than treating the database as a sidecar file that can much more easily become disconnected with one or more top-level directories it ostensibly describes the contents of.
Here seems to be another SMALL undocumented dependency of this change: completing the /usrmove thing to cover the whole world including /opt, /etc, /var, and presumably /boot as well because packages put stuff in it.
Rpm wont care if you want to move content from /etc to /usr/etc and ditto for /var, /opt and /boot which is also used by packages, ie to complete to complete the great /usrmove thing started a decade ago. But that's quite a hidden agenda in this change if you ask me.
This change proposal is narrow in scope. It imagines a future snapshot+rollback regime of some kind, but isn't depending on any particular file system or way of achieving it.
I personally like the transactional updates approach. I like what rpm-ostree is doing, updating the system root out of band rather than updating the currently running system. I like the idea these updates can happen in a container, if they fail for any reason we just remove that bad tree - it's a completely disposable thing until it successfully updates and maybe we can even automate some simple tests (it gets to X milestone in the boot process) before we activate it as the current root. I like the idea of users being able to follow the trail of breadcrumbs of how systems boot. My strong bias from many years in Fedora QA is we deal every cycle with boot/startup failure in some form or another, and the less obscure less fragile more self-describing and standard that process can be, the easier it is to avoid, find and fix problems.
We also know that (open)SUSE has proven a working boot to snapshot design. It's just complicated. They're the first to admit it, and that they've learned a lot from the experience. Things have changed in 10 years. I imagine we're wandering in a very big library and we don't know exactly what book we're looking for, but we're perusing with a purpose. If there is a hidden agenda, it's because none of the change owners are attached to any one particular boot to snapshot design. We know it's not going to look identical to anything else out there because it has to work with a purpose for Fedora, but we also don't know the details of what that's going to look like.
-- Chris Murphy
Dne 12. 01. 22 v 22:05 Chris Murphy napsal(a):
On Wed, Jan 12, 2022 at 2:04 AM Panu Matilainen pmatilai@redhat.com wrote:
On 1/12/22 10:45, Chris Murphy wrote:
On Tue, Jan 11, 2022 at 2:00 AM Panu Matilainen pmatilai@redhat.com wrote:
For many practical purposes it's probably just rearranging the chairs, but a separate top-level directory describing the *system* state seems instinctively *much* more correct solution to it than stuffing it somewhere deep inside a loosely related fs.
If rpm is constrained to /usr, then its database can properly be somewhere in /usr
If rpm is unconstrained, with transactions touching any of /usr /var /opt /etc, the resulting paradigm is inherently complicated, and an additional top-level directory doesn't help make it less complicated.
WTF is this talk about constraining? There's no such constraint in rpm, and never will.
I'm not talking about changing the nature of the beast. I'm talking about how it's used as a tool in a distribution with a lot of other requirements and tradeoffs.
Expanding rpm is also a valid path. We have an example "DNF snapper plug-in" that demonstrates at least DNF *could* get into the business of managing snapshots and rollbacks, if it wants that responsibility, if it's helpful and worth the effort, and if the resources exist to do it. That's an interesting discussion.
And Fedora packages are not constrained there either, a vast percentage places stuff in /etc, touches directory tree in /var and then there's /boot and the software collections and third party software.
/boot might need its own manager(s). I'm strongly leaning toward bootupd and fwupd being the only things that touch /boot and everything else is proscribed.
/boot is a small but hot mess right now. RPM, grub, grubby, dracut, all can touch /boot and in the Boot Loader Spec context there's multiple OS's sharing /boot. About the only thing I like better is linux as the bootloader, e.g. Linux Boot, so we have real device and file system drivers. Anything that manages /boot needs to needs understand snapshots, and which kernel+initramfs can boot which snapshots.
Either top-level directories have their own databases, so rpm knows what's in them even if one is rolled back but not another; or there's one database to describe all of these top-level directories which then need to share a single (sub)volume so they're all snapshot and rolled back together. Either way, there's additionally a need for carve outs for things that shouldn't be rolled back.
Erm. Packages don't live conveniently inside a single top level tree, they can't be split up that way.
Right. I view this as a criticism, not a benefit though. Neither rpm nor the FHS are organizing things in a way that helps us do rollbacks. They're impediments we keep having to take other shortcuts and workarounds to achieve that goal, and as yet they still have too many shortcomings. It really isn't to be underestimated the importance of multiple independent parties coming to the same kinds of conclusions.
Given the choice, I prefer rpm only touches /usr, which includes /usr/var and /usr/etc.
But if left unconstrained, having databases inside the directories they describe is more reliable than treating the database as a sidecar file that can much more easily become disconnected with one or more top-level directories it ostensibly describes the contents of.
Here seems to be another SMALL undocumented dependency of this change: completing the /usrmove thing to cover the whole world including /opt, /etc, /var, and presumably /boot as well because packages put stuff in it.
Rpm wont care if you want to move content from /etc to /usr/etc and ditto for /var, /opt and /boot which is also used by packages, ie to complete to complete the great /usrmove thing started a decade ago. But that's quite a hidden agenda in this change if you ask me.
This change proposal is narrow in scope. It imagines a future snapshot+rollback regime of some kind, but isn't depending on any particular file system or way of achieving it.
I personally like the transactional updates approach. I like what rpm-ostree is doing, updating the system root out of band rather than updating the currently running system.
Actually, shouldn't rpm-ostree carry around some copy of the RPM database, which would describe the state of /usr and once the update is successful (or snapshot active?), merge it into the main system RPM database? Apparently, something like this is already happening e.g. for /etc content if I understand it correctly. Or this is the direction in which the /boot will be handled eventually. Or possibly it could be just some linked (possibly just read only) database.
IOW, imagine, that we still keep the system read/write RPM database in /var and we teach RPM to use additional database(s). So RPM would know that there might be some additional database e.g. in /usr/var/ ... The system database would not know nothing about content of /usr, but once /usr was mounted, `rpm -q` would provide the information from the linked RPM database.
Vít
I like the idea these updates can happen in a container, if they fail for any reason we just remove that bad tree - it's a completely disposable thing until it successfully updates and maybe we can even automate some simple tests (it gets to X milestone in the boot process) before we activate it as the current root. I like the idea of users being able to follow the trail of breadcrumbs of how systems boot. My strong bias from many years in Fedora QA is we deal every cycle with boot/startup failure in some form or another, and the less obscure less fragile more self-describing and standard that process can be, the easier it is to avoid, find and fix problems.
We also know that (open)SUSE has proven a working boot to snapshot design. It's just complicated. They're the first to admit it, and that they've learned a lot from the experience. Things have changed in 10 years. I imagine we're wandering in a very big library and we don't know exactly what book we're looking for, but we're perusing with a purpose. If there is a hidden agenda, it's because none of the change owners are attached to any one particular boot to snapshot design. We know it's not going to look identical to anything else out there because it has to work with a purpose for Fedora, but we also don't know the details of what that's going to look like.
-- Chris Murphy _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Thu, Jan 13, 2022, at 7:52 AM, Vít Ondruch wrote:
Actually, shouldn't rpm-ostree carry around some copy of the RPM database, which would describe the state of /usr and once the update is successful (or snapshot active?), merge it into the main system RPM database? Apparently, something like this is already happening e.g. for /etc content if I understand it correctly. Or this is the direction in which the /boot will be handled eventually. Or possibly it could be just some linked (possibly just read only) database.
I think you are confusing/conflating image based updates and client side snapshots (and relatedly, client side offline updates, which is what https://github.com/OpenSUSE/transactional-update). They are related, but different.
It's important to emphasize that rpm-ostree systems are by default, *image based*. For example, every single Fedora CoreOS system running the latest "stable" commit has *bit for bit identical /usr*, including the RPM database in /usr/share/rpm.
All SELinux labeling was computed server side. All %post scripts were run on the build server.
We perform integration testing on that image before it ever hits any user's disk. See https://github.com/coreos/fedora-coreos-tracker/issues/1066 for example. (This integration testing aspect is not true of Silverblue or IoT, because they just ship what bodhi ships, which is its own big problem)
If one chooses to engage client side layering (or overrides), rpm-ostree will (each time an upgrade or change is performed) indeed regenerate the rpm database using the "pristine" base image's version of the rpmdb.
What you are talking about seems more related to client side snapshots and/or client side offline updates. So in your phrasing above I'd say something more like "proposed dnf/snapper tooling" or so, and not rpm-ostree which works in a different way.
IOW, imagine, that we still keep the system read/write RPM database in /var and we teach RPM to use additional database(s). So RPM would know that there might be some additional database e.g. in /usr/var/ ... The system database would not know nothing about content of /usr, but once /usr was mounted, `rpm -q` would provide the information from the linked RPM database.
I am having trouble understanding what you are thinking here. If you are interested in this, I'd recommend taking some time trying out an existing system that is doing some of these things; either an OpenSUSE transactional-update system or an rpm-ostree system for example.
Dne 13. 01. 22 v 15:26 Colin Walters napsal(a):
On Thu, Jan 13, 2022, at 7:52 AM, Vít Ondruch wrote:
Actually, shouldn't rpm-ostree carry around some copy of the RPM database, which would describe the state of /usr and once the update is successful (or snapshot active?), merge it into the main system RPM database? Apparently, something like this is already happening e.g. for /etc content if I understand it correctly. Or this is the direction in which the /boot will be handled eventually. Or possibly it could be just some linked (possibly just read only) database.
I think you are confusing/conflating image based updates and client side snapshots (and relatedly, client side offline updates, which is what https://github.com/OpenSUSE/transactional-update). They are related, but different.
I think you are doing unnecessary difference between those.
In my view, there is no difference if /usr is read-only, network mounted, expanded from tarball or managed by rpm-ostree. The point is that the location, while being available on the user system, is not modifiable by RPM. The only requirement is that RPM is be able to provides metadata about that location. In this case, it makes perfect sense if the /usr contains some metadata in RPM consumable format (be it RPM database, but that is technical detail, because it could be its simpler version for RO filesystems).
Now should there be different locations really managed by RPM, such as /opt, there is no reason why there should not be the system RPM database, which would store location about that parts of the system. To cover this scenario for rpm-ostree, there needs to be done something complex, in your words:
If one chooses to engage client side layering (or overrides), rpm-ostree will (each time an upgrade or change is performed) indeed regenerate the rpm database using the "pristine" base image's version of the rpmdb.
And I say this is wrong concept (understandably given by technical limitation). RPM in this case should keep using its system database and understand that /usr was delivered by different means, it can't manage the /usr content, but there is the metadata available.
In this case, when the base image was updated, the RPM database would not need to be regenerated. However, the base image update could be clocked when e.g. RPM discovered, that it would broke the system dependencies.
Vít
On Wed, Jan 12, 2022 at 9:45 AM Chris Murphy lists@colorremedies.com wrote:
Given the choice, I prefer rpm only touches /usr, which includes /usr/var and /usr/etc.
I must say that, of all the bad suggestions in the multiple threads this proposal has spawned, I dislike using "/usr/var/" the least.
The path "/usr/lib/sysimage/rpm" does look very out-of-place in non-image-based systems, so *if* we want to move the rpmdb to a place that's consistent across all our Editions, it should also be a location name that makes sense across all Editions.
A path that contains "sysimage" conveys to me that this is somehow only related to image based variants. Something under /usr/var would at least be an generic, non-confusing path.
Fabio
On Friday, 14 January 2022 at 00:05, Fabio Valentini wrote:
On Wed, Jan 12, 2022 at 9:45 AM Chris Murphy lists@colorremedies.com wrote:
Given the choice, I prefer rpm only touches /usr, which includes /usr/var and /usr/etc.
I must say that, of all the bad suggestions in the multiple threads this proposal has spawned, I dislike using "/usr/var/" the least.
The path "/usr/lib/sysimage/rpm" does look very out-of-place in non-image-based systems, so *if* we want to move the rpmdb to a place that's consistent across all our Editions, it should also be a location name that makes sense across all Editions.
A path that contains "sysimage" conveys to me that this is somehow only related to image based variants. Something under /usr/var would at least be an generic, non-confusing path.
+1 to that. /usr/lib/sysimage is definitely a non-obvious location for rpmdb, while /usr/var/lib at least suggests it's something akin to /var/lib.
Regards, Dominik
On Thu, Jan 13, 2022, at 6:05 PM, Fabio Valentini wrote:
The path "/usr/lib/sysimage/rpm" does look very out-of-place in non-image-based systems, so *if* we want to move the rpmdb to a place that's consistent across all our Editions, it should also be a location name that makes sense across all Editions.
I don't think we should discount alignment with OpenSUSE. When they originally proposed /usr/lib/sysimage I started to write a bikeshed reply email (like many that have been posted here) around why /usr/share was a bit better but then I said to myself: "You know what? I don't care as long as we get it in /usr. Since they're driving the change, and there really isn't any technical compelling reason to do something else, it's fine."
Also, proliferation of these paths has a cost; see e.g. https://github.com/openSUSE/libsolv/pull/386 Though in practice *most* cases will be fine just chasing a symlink from /var/lib/rpm.
On Fri, Jan 14, 2022 at 7:16 PM Colin Walters walters@verbum.org wrote:
On Thu, Jan 13, 2022, at 6:05 PM, Fabio Valentini wrote:
The path "/usr/lib/sysimage/rpm" does look very out-of-place in non-image-based systems, so *if* we want to move the rpmdb to a place that's consistent across all our Editions, it should also be a location name that makes sense across all Editions.
I don't think we should discount alignment with OpenSUSE. When they originally proposed /usr/lib/sysimage I started to write a bikeshed reply email (like many that have been posted here) around why /usr/share was a bit better but then I said to myself: "You know what? I don't care as long as we get it in /usr. Since they're driving the change, and there really isn't any technical compelling reason to do something else, it's fine."
Also, proliferation of these paths has a cost; see e.g. https://github.com/openSUSE/libsolv/pull/386 Though in practice *most* cases will be fine just chasing a symlink from /var/lib/rpm.
Wait, I thought this change was about making the path consistent within Fedora variants? I understand that converging on the same path as OpenSUSE makes sense, but does that mean we should not consider if there's a better alternative? ... And the "sysimage" path makes even less sense to me in the OpenSUSE context, since they don't have an OSTree based variant at all (unless I am mistaken)?
Fabio
On Fri, Jan 14, 2022 at 3:51 PM Fabio Valentini decathorpe@gmail.com wrote:
On Fri, Jan 14, 2022 at 7:16 PM Colin Walters walters@verbum.org wrote:
On Thu, Jan 13, 2022, at 6:05 PM, Fabio Valentini wrote:
The path "/usr/lib/sysimage/rpm" does look very out-of-place in non-image-based systems, so *if* we want to move the rpmdb to a place that's consistent across all our Editions, it should also be a location name that makes sense across all Editions.
I don't think we should discount alignment with OpenSUSE. When they originally proposed /usr/lib/sysimage I started to write a bikeshed reply email (like many that have been posted here) around why /usr/share was a bit better but then I said to myself: "You know what? I don't care as long as we get it in /usr. Since they're driving the change, and there really isn't any technical compelling reason to do something else, it's fine."
Also, proliferation of these paths has a cost; see e.g. https://github.com/openSUSE/libsolv/pull/386 Though in practice *most* cases will be fine just chasing a symlink from /var/lib/rpm.
Wait, I thought this change was about making the path consistent within Fedora variants?
CoreOS/Silverblue/Kinoite ls -l /var/lib/rpm lrwxrwxrwx. 1 root root 19 Dec 6 12:32 /var/lib/rpm -> ../../usr/share/rpm $ ls -l /usr/share/rpm total 52684 -rw-r--r--. 3 root root 53915648 Dec 31 1969 rpmdb.sqlite -rw-r--r--. 5 root root 32768 Jan 14 18:10 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Jan 11 18:51 rpmdb.sqlite-wal $ ls -l /usr/lib/sysimage/rpm lrwxrwxrwx. 3 root root 15 Dec 6 12:23 /usr/lib/sysimage/rpm -> ../../share/rpm
This symlink is present on Silverblue but I don't see it on a stock CoreOS image before it's been booted and subject to setup by ignition. $ ls -l /var/lib/rpm -> ../../usr/share/rpm
I understand that converging on the same path as OpenSUSE makes sense, but does that mean we should not consider if there's a better alternative? ...
I think it's fine to consider alternatives, but the tradeoff might be that (open)SUSE sticks with what they've got. I have no idea but that could be a question posed to them before deciding this.
And the "sysimage" path makes even less sense to me in the OpenSUSE context, since they don't have an OSTree based variant at all (unless I am mistaken)?
Absent the rpmdb, and ignoring all other considerations like storage and performance cost, could RPM learn to reconstruct the rpmdb? What would it need locally to do that? Would it need to keep rpm package headers and compare to what's installed? Some variation on rpm -Va?
The point would be to keep /var/lib/rpm/rpmdb* allow it to be disposable, but with consequences. If rpm packages can be disembodied from their payload, just leaving the metadata, seems like it's plausible to reconstruct an rpmdb from checksums of what's installed versus would could be installed. This raises questions like, garbage collection of all these rpm files, would they be removed when the package is removed?
-- Chris Murphy
Am 14.01.2022 um 23:51 schrieb Fabio Valentini decathorpe@gmail.com:
Wait, I thought this change was about making the path consistent within Fedora variants?
The question still is whether this is actually useful and beneficial.
All the arguments for this move that I have read so far explain benefits related to an image based distribution. I have not seen any advantages of this move for a file based distribution like Server or Workstation. And when I look at tools like LVM snapshot, I can't see any either. And I couldn’t find something in a short online search. But maybe I missed something? Then I would be grateful for some info/links.
In the case that there is no benefit for file based distributions, they only suffer the "malefit" of losing FHS compliance and possibly having to adapt administration scripts and routines. Then it would make more sense to move the RPM database only for image based distributions (which has obviously already been done anyway).
Peter
On Sun, Jan 16, 2022 at 5:59 PM Peter Boy pboy@uni-bremen.de wrote:
Am 14.01.2022 um 23:51 schrieb Fabio Valentini decathorpe@gmail.com:
Wait, I thought this change was about making the path consistent within Fedora variants?
The question still is whether this is actually useful and beneficial.
All the arguments for this move that I have read so far explain benefits related to an image based distribution. I have not seen any advantages of this move for a file based distribution like Server or Workstation. And when I look at tools like LVM snapshot, I can't see any either. And I couldn’t find something in a short online search. But maybe I missed something? Then I would be grateful for some info/links.
It benefits LVM-based OS snapshotting equally well. However, the tooling for LVM snapshots are under-developed, so it's not used so much for this.
Once upon a time, Neal Gompa ngompa13@gmail.com said:
It benefits LVM-based OS snapshotting equally well. However, the tooling for LVM snapshots are under-developed, so it's not used so much for this.
How? Since the standard install uses a single root filesystem that includes /usr and /var, I can't see any benefit to it there either.
On Sun, Jan 16, 2022 at 6:47 PM Chris Adams linux@cmadams.net wrote:
Once upon a time, Neal Gompa ngompa13@gmail.com said:
It benefits LVM-based OS snapshotting equally well. However, the tooling for LVM snapshots are under-developed, so it's not used so much for this.
How? Since the standard install uses a single root filesystem that includes /usr and /var, I can't see any benefit to it there either.
It does *now*, but it's not necessarily desirable to keep it that way if you want this capability. For Btrfs, shuffling subvolumes is cheap. For LVM, shuffling LVs is a bit more work.
Once upon a time, Neal Gompa ngompa13@gmail.com said:
On Sun, Jan 16, 2022 at 6:47 PM Chris Adams linux@cmadams.net wrote:
Once upon a time, Neal Gompa ngompa13@gmail.com said:
It benefits LVM-based OS snapshotting equally well. However, the tooling for LVM snapshots are under-developed, so it's not used so much for this.
How? Since the standard install uses a single root filesystem that includes /usr and /var, I can't see any benefit to it there either.
It does *now*, but it's not necessarily desirable to keep it that way if you want this capability. For Btrfs, shuffling subvolumes is cheap. For LVM, shuffling LVs is a bit more work.
I know about LVM snapshots - I used them for years to get point-in-time backups. But that doesn't explain a benefit to splitting /, /usr, and /var into separate filesystems/LVs/whatever.
On Sun, Jan 16, 2022 at 3:59 PM Peter Boy pboy@uni-bremen.de wrote:
Am 14.01.2022 um 23:51 schrieb Fabio Valentini decathorpe@gmail.com:
Wait, I thought this change was about making the path consistent within Fedora variants?
The question still is whether this is actually useful and beneficial.
If you value Fedora having a snapshot and rollback scheme of some kind, it's useful and beneficial. If you don't, then the change is neutral because it has not a single technical downside presented so far - just emotive ones.
This change also doesn't put pressure on future decisions. It doesn't favor particular ways of snapshotting or rollback mechanisms, doesn't care about what volume manager is being used, etc.
All the arguments for this move that I have read so far explain benefits related to an image based distribution. I have not seen any advantages of this move for a file based distribution like Server or Workstation.
Again if you see no value in snapshots/rollbacks, you don't see the advantage. If you like the idea, then you'd also necessarily come to realize that some pressure on organizing files into locations with compatible life cycles, so those locations can be independently rolled back. The change is already employed in traditional RPM based (open)SUSE a couple of years, the links to those discussions are in the change proposal.
And when I look at tools like LVM snapshot, I can't see any either. And I couldn’t find something in a short online search. But maybe I missed something? Then I would be grateful for some info/links.
If anything it benefits LVM more, because each additional carveout on LVM requires a separate file system. On btrfs it's cheap to add subvolumes since they take up no space and instead all share a single storage pool, though it organizationally adds complexity.
Would putting /var/lib/rpm on a dedicated LVM LV actually be entertained? I think it wouldn't.
Once upon a time, Chris Murphy lists@colorremedies.com said:
If you value Fedora having a snapshot and rollback scheme of some kind, it's useful and beneficial. If you don't, then the change is neutral because it has not a single technical downside presented so far - just emotive ones.
It's only beneficial for snapshots if you believe that only /usr matters, which I don't believe is true. It is not neutral, because it does change things from long-standing defaults.
It's also a change away from upstream, which normally I thought Fedora preferred to avoid unless there's a significantly compelling reason.
On Sun, Jan 16, 2022 at 9:36 PM Chris Adams linux@cmadams.net wrote:
Once upon a time, Chris Murphy lists@colorremedies.com said:
If you value Fedora having a snapshot and rollback scheme of some kind, it's useful and beneficial. If you don't, then the change is neutral because it has not a single technical downside presented so far - just emotive ones.
It's only beneficial for snapshots if you believe that only /usr matters, which I don't believe is true.
It might be true that it's the only one that should matter, if you care to have the benefit of a clear cut snapshot and rollback design. This is the /etc/fstab for (open)SUSE Tumbleweed.
$mainfsuuid / btrfs defaults 0 0 $mainfsuuid /var btrfs subvol=/@/var 0 0 $mainfsuuid /usr/local btrfs subvol=/@/usr/local 0 0 $mainfsuuid /srv btrfs subvol=/@/srv 0 0 $mainfsuuid /root btrfs subvol=/@/root 0 0 $mainfsuuid /opt btrfs subvol=/@/opt 0 0 $mainfsuuid /home btrfs subvol=/@/home 0 0 $mainfsuuid /boot/grub2/x86_64-efi btrfs subvol=/@/boot/grub2/x86_64-efi 0 0 $mainfsuuid /boot/grub2/i386-pc btrfs subvol=/@/boot/grub2/i386-pc 0 0 $mainfsuuid /.snapshots btrfs subvol=/@/.snapshots 0 0 $swapfsuuid swap swap defaults 0 0 $espfsuuid /boot/efi vfat utf8 0 2
It lacks a separate line for /var/tmp/rpm, which it would otherwise need, because the rpmdb is in /usr.
It looks like this because /usr isn't the only thing that's important. These carve outs exist because it was easier to do with Btrfs subvolumes (effectively bind mounts behind the scenes) than (a) with individual file systems or (b) committing to reorganizing the filesystem hierarchy. Now (open)SUSE is doing the hard work to reorganize the filesystem hierarchy where RPM only touches /usr.
If we're feeling bold and maybe even brazen, maybe we'd pull everything out of /usr /var /etc that can be called 'the system'. Specifically, that which is system critical for the purpose of booting and startup. Things that get us to a working login prompt, but not beyond that (or at least not much beyond it). Stick all of that into a new top-level directory called "system". Within that could be usr/ var/ etc/ if you want. This "system" subvolume or LV is what would be subject to snapshot and rollback.
This could be done with metadata defining what individual files are "system critical" so that 'dnf history rollback' could do a fine grained revert for such those files. But that presumes we can even boot to a system sufficiently working that this command could succeed. I think it's better to have a system that's prepared for the possibility an update can introduce trouble, and has the ability to detect failure and reboot to a previous known working state.
On Sunday, January 16, 2022 11:16:57 PM EST Chris Murphy wrote:
On Sun, Jan 16, 2022 at 3:59 PM Peter Boy pboy@uni-bremen.de wrote:
Am 14.01.2022 um 23:51 schrieb Fabio Valentini decathorpe@gmail.com:
Wait, I thought this change was about making the path consistent within Fedora variants?
The question still is whether this is actually useful and beneficial.
If you value Fedora having a snapshot and rollback scheme of some kind, it's useful and beneficial. If you don't, then the change is neutral because it has not a single technical downside presented so far - just emotive ones.
I'm a little concerned about what this increased writing to /usr, which many people have on a SSD, will do. I have everything that gets written to with any regularity on spinning disks. Everything else is SSD.
I have to agree with Chris Adams that rpms own things on many top level directories. I have rpms that own programs/files that live on /opt and other top level directories. I also agree that most programs cannot recreate their directory paths and consider it's absense to be a catastrophic failure.
I also see things like kmail which uses mariadb as it's storage. It occassionally contains migration scripts to change the format of the database. It never has a migration script to go backwards. We do the same thing with fapolicyd and its lmdb backing storage. We migrate forward, but we never allow backwards migration. To roll back fapolicyd, you'd need to snapshot /var and roll it back. But since /var has the mail spool and other accumulated data, you'd risk losing lots of stuff if /var was rolled back.
I'd think the solution for image based distros is to put the rpmdb in /usr/ lib/rpm and bind mount it to /var/lib/rpm. Doing it this way means no changes are needed.
-Steve
Am 17.01.2022 um 05:16 schrieb Chris Murphy lists@colorremedies.com:
On Sun, Jan 16, 2022 at 3:59 PM Peter Boy pboy@uni-bremen.de wrote:
Am 14.01.2022 um 23:51 schrieb Fabio Valentini decathorpe@gmail.com:
Wait, I thought this change was about making the path consistent within Fedora variants?
The question still is whether this is actually useful and beneficial.
If you value Fedora having a snapshot and rollback scheme of some kind, it's useful and beneficial. If you don't, then the change is neutral because it has not a single technical downside presented so far - just emotive ones.
The loss of LSB / FHS compliance is by no means "just emotive", nor is the consequence of probably having to adapt an unknown number of scripts, backup routines, etc. .
As you stated in a previous post, LVM snapshot is „under-developed“ and therefore currently not useful. Is there another snapshot & rollback tool in Fedora repository I can use out of the box (after I moved rpm db to /usr)? Unfortunately, I don’t know one. Currently I would have to create a LVM snapshot and a manually crafted selective backup of files and subdirectories. And test all the stuff. Not particularly practical.
I guess, an achievement of a truly viable snapshot and rollback system for file-based distributions requires far a greater number of relocations than just the rpm db. The article of a (presumably) Suse employee I’m trying to retrieve (see an earlier post of mine) offers a proposal for this very purpose (including a backwards compatibility link system). This would eventually end up in an FHS 4.x.
Moving the rpm db alone adds only disadvantages to file-based distributions, not a single advantage. And it is not neutral either.
In the end, we don't really need to do anything. If I understand correctly, rpm-ostree is already implementing the change anyway, without any change voting, and everyone else will continue as before, following LSB/FHS and the current Fedora guidelines.
Again if you see no value in snapshots/rollbacks, you don't see the advantage. If you like the idea, then you'd also necessarily come to realize that some pressure on organizing files into locations with compatible life cycles, so those locations can be independently rolled back.
I see a lot of value in snapshot&rollback systems. And I see not only „some“ pressure (or better requirements) to reorganise files into different locations. The rpm db is far from sufficient. The proposal mentioned above did exactly that, but rpm db did not end up in /usr but together with parts of /var and /etc somewhere else I don’t remember in detail.
And if we make adjustments to achieve a snapshot/rollback system, then we should do it right and not stop halfway unfinished.
Peter
On Fri, Jan 14, 2022 at 5:51 PM Fabio Valentini decathorpe@gmail.com wrote:
On Fri, Jan 14, 2022 at 7:16 PM Colin Walters walters@verbum.org wrote:
On Thu, Jan 13, 2022, at 6:05 PM, Fabio Valentini wrote:
The path "/usr/lib/sysimage/rpm" does look very out-of-place in non-image-based systems, so *if* we want to move the rpmdb to a place that's consistent across all our Editions, it should also be a location name that makes sense across all Editions.
I don't think we should discount alignment with OpenSUSE. When they originally proposed /usr/lib/sysimage I started to write a bikeshed reply email (like many that have been posted here) around why /usr/share was a bit better but then I said to myself: "You know what? I don't care as long as we get it in /usr. Since they're driving the change, and there really isn't any technical compelling reason to do something else, it's fine."
Also, proliferation of these paths has a cost; see e.g. https://github.com/openSUSE/libsolv/pull/386 Though in practice *most* cases will be fine just chasing a symlink from /var/lib/rpm.
Wait, I thought this change was about making the path consistent within Fedora variants? I understand that converging on the same path as OpenSUSE makes sense, but does that mean we should not consider if there's a better alternative? ... And the "sysimage" path makes even less sense to me in the OpenSUSE context, since they don't have an OSTree based variant at all (unless I am mistaken)?
openSUSE originally did the move because standard openSUSE has a snapshot+rollback scheme and tracking the rpmdb is straightforward in /usr with all the other system state data. This benefited them for the development and release of openSUSE MicroOS a couple years later.
Am 17.01.2022 um 00:09 schrieb Neal Gompa ngompa13@gmail.com:
On Fri, Jan 14, 2022 at 5:51 PM Fabio Valentini decathorpe@gmail.com wrote:
….
openSUSE originally did the move because standard openSUSE has a snapshot+rollback scheme and tracking the rpmdb is straightforward in /usr with all the other system state data. This benefited them for the development and release of openSUSE MicroOS a couple years later.
@Neal, I remember a Suse employee made once (about a year ago) a proposal to slightly modify FHS to better separate distro specific from local specific content, especially in /etc. Unfortunately I can't find the link again. Do you happen to know if that was related to Suse's snapshot+rollback scheme?
Thanks Peter
On Sun, Jan 16, 2022 at 4:34 PM Peter Boy pboy@uni-bremen.de wrote:
Am 17.01.2022 um 00:09 schrieb Neal Gompa ngompa13@gmail.com:
On Fri, Jan 14, 2022 at 5:51 PM Fabio Valentini decathorpe@gmail.com wrote:
….
openSUSE originally did the move because standard openSUSE has a snapshot+rollback scheme and tracking the rpmdb is straightforward in /usr with all the other system state data. This benefited them for the development and release of openSUSE MicroOS a couple years later.
@Neal, I remember a Suse employee made once (about a year ago) a proposal to slightly modify FHS to better separate distro specific from local specific content, especially in /etc. Unfortunately I can't find the link again. Do you happen to know if that was related to Suse's snapshot+rollback scheme?
Maybe this: https://github.com/lnussel/lnussel.github.io/blob/fs/_posts/2020-12-16-fslay...
Am 17.01.2022 um 05:17 schrieb Chris Murphy lists@colorremedies.com:
On Sun, Jan 16, 2022 at 4:34 PM Peter Boy pboy@uni-bremen.de wrote:
@Neal, I remember a Suse employee made once (about a year ago) a proposal to slightly modify FHS to better separate distro specific from local specific content, especially in /etc. Unfortunately I can't find the link again. Do you happen to know if that was related to Suse's snapshot+rollback scheme?
Maybe this: https://github.com/lnussel/lnussel.github.io/blob/fs/_posts/2020-12-16-fslay...
Yes, that the right direction. I’ll start my search from that.
Thanks Peter
On Tue, Jan 11, 2022 at 11:00:28AM +0200, Panu Matilainen wrote:
On 1/10/22 23:53, Chris Murphy wrote:
On Mon, Jan 10, 2022 at 9:20 AM David Cantrell dcantrell@redhat.com wrote:
On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
== Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE. [snip]
Moving the RPM database to /usr feels incorrect to me, but we should move it to gain the improvements as noted in the feature proposal.
Going back to the original discussions on moving rpmdb...
Preferred is a new top-level location in /usr, .e.g /usr/sysimage/rpm. Next is "least worst" in /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006764.html http://lists.rpm.org/pipermail/rpm-maint/2017-October/006722.html
And the convergence was on /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006785.html
I don't see how /state solves the problem, rather than just rearranging the chairs.
The problem with /usr/something is that the rpmdb is not specific to /usr contents at all, and unlike any other content in there, so putting it there just *feels so wrong*. That's what /state or /sysimage or, as we now have, /var supposedly solves.
+1
I thought I'd suggested something at / level back then (in https://lists.rpm.org/pipermail/rpm-maint/2017-October/006697.html and/or https://lists.rpm.org/pipermail/rpm-maint/2017-October/006699.html) but seems like memory is failing me :) Maybe I thought it would seem too outrageous to FHS believers to bother.
The point was though, that the rpmdb is not at all the only data of this kind and so having a dedicated home makes sense.
For many practical purposes it's probably just rearranging the chairs, but a separate top-level directory describing the *system* state seems instinctively *much* more correct solution to it than stuffing it somewhere deep inside a loosely related fs.
Also +1
Just FWIW, I would quit my whining about this right there if it went to a new toplevel directory instead because it just *feels* right unlike /usr.
Numerous followups have noted the requirement that /usr contain read-only content, that it be shareable across hosts, and similar concepts. While this may or may not doable now like we could in the past, the bigger thing to me is around the understanding of what /usr contains. It is generally understood that /usr contains [most of] the installed system. What I think is a bigger requirement or expection now is that one can tar up /usr and transport it to another system or virtual machine or container and expect that it will _probably_ work maybe with a bit of tinkering. This is a really valuable thing to have for developers. Moving the RPM database to this tree adds a component that is unnecessary and sort of out of place.
Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt?
If /usr is to be truly portable and have e.g. 'rpm query, verify, remove, reinstall' work as expected, you need the metadata (the database) representing its state to always come along for the ride. Either the database is already in /usr, or you have to make sure /usr and /state are inseparable.
If /usr and /state are inseparable, and if rpm can also describe anything in /etc or /var or /opt, then all or part of those directories are also inseparable from /state. And thus /usr. So I think /state doesn't help.
For one, /state (or whatever toplevel directory) allows for the fact that there are write-operations to rpmdb that do not touch any external files while maintaining read-only /usr. Such as rpmdb --rebuilddb, or rpm --import.
And like mentioned in the original discussion and now here, although the discussion is on rpmdb, it's not the only data of this kind.
Indeed. We have an opportunity here to get data that fits this categorization in to a more correct location on the system. rpmdb is a big one, but certainly not the only one.
To what degree do rpm and dnf intend to touch locations outside of /usr *and care* about tracking those changes?
I don't understand the question. Rpm tracks and cares about all content it knows about equally, regardless of the path. /usr is NOT special in any way to rpm, it's just that most of *distro* content ends up in there but a huge number of packages have content spread across /etc too.
Yes, the special semantics around /usr is from the rpm-ostree side and not rpm.
I think rpm can't remain static for all time. It either needs to become aware of multiple root trees, and even mix and match top-level directories to create variable roots. And possibly even manage these things. Or it needs to constrain its reach to /usr and /opt. Whether /usr and /opt are tied together, or can be mix and match with their own rpmdb's, I have no strong opinion on.
Oh, multiple rpmdbs. It's a while since that last turned up. It gets tossed around as a solution but to me it looks like it brings more problems than it solves.
On Mon, Jan 10, 2022 at 02:53:52PM -0700, Chris Murphy wrote:
Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt?
We have three similar locations: /usr (system vendor tree), /usr/local (admin non-packages installations), /opt (external vendor tree). In other words, both /usr and /opt are for packages, but from different sources. As an admin, you'd want to treat both package types the same, and e.g. roll them back together. So having a separate tree for /opt doesn't make much sense.
[At some point in the future] /opt should be renamed to /usr/opt and symlinked for backwards compat.
If /usr is to be truly portable and have e.g. 'rpm query, verify, remove, reinstall' work as expected, you need the metadata (the database) representing its state to always come along for the ride. Either the database is already in /usr, or you have to make sure /usr and /state are inseparable.
If /usr and /state are inseparable, and if rpm can also describe anything in /etc or /var or /opt, then all or part of those directories are also inseparable from /state. And thus /usr. So I think /state doesn't help.
Yeah, /state doesn't help with anything. It'd be just some part of the whole system state, without a clear separation of why this particular subset. My preference: /usr/lib/sysimage/rpm > /var/lib/rpm > /state.
To what degree do rpm and dnf intend to touch locations outside of /usr *and care* about tracking those changes?
Traditionally, packages installed all kinds of files all over the place. But we're slowly and painfully moving towards the model where: 1. packages are only allowed to install under /usr, /var, and /etc. (Or under /opt, but I'd want to move that to /usr/opt…) 2. packages must support /var/cache being wiped at any time, and most packages support anything under /var being wiped at any time. 3. systemd and other projects are trying to only use /etc for local admin state, and support "factory reset" by wiping /etc and /var.
So although rpm manages files under /var and /etc, it is not to the same extent as /usr. In this light it makes a lot of sense to hold the rpmdb state under /usr/ somewhere. The exact subdirectory doesn't really matter, it's just a matter of convention, so obviously we want to follow what opensuse and others are already using.
Zbyszek
On 1/12/22 11:05, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Jan 10, 2022 at 02:53:52PM -0700, Chris Murphy wrote:
Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt?
We have three similar locations: /usr (system vendor tree), /usr/local (admin non-packages installations), /opt (external vendor tree). In other words, both /usr and /opt are for packages, but from different sources. As an admin, you'd want to treat both package types the same, and e.g. roll them back together. So having a separate tree for /opt doesn't make much sense.
[At some point in the future] /opt should be renamed to /usr/opt and symlinked for backwards compat.
If /usr is to be truly portable and have e.g. 'rpm query, verify, remove, reinstall' work as expected, you need the metadata (the database) representing its state to always come along for the ride. Either the database is already in /usr, or you have to make sure /usr and /state are inseparable.
If /usr and /state are inseparable, and if rpm can also describe anything in /etc or /var or /opt, then all or part of those directories are also inseparable from /state. And thus /usr. So I think /state doesn't help.
Yeah, /state doesn't help with anything. It'd be just some part of the whole system state, without a clear separation of why this particular subset. My preference: /usr/lib/sysimage/rpm > /var/lib/rpm > /state.
To what degree do rpm and dnf intend to touch locations outside of /usr *and care* about tracking those changes?
Traditionally, packages installed all kinds of files all over the place. But we're slowly and painfully moving towards the model where:
- packages are only allowed to install under /usr, /var, and /etc. (Or under /opt, but I'd want to move that to /usr/opt…)
- packages must support /var/cache being wiped at any time, and most packages support anything under /var being wiped at any time.
- systemd and other projects are trying to only use /etc for local admin state, and support "factory reset" by wiping /etc and /var.
Oh, right. More hidden agenda behind this thing. When looking at it with these glasses on, it explains quite a few things about the change proposal, such as completely ignoring the fact that nearly all packages put something in /etc.
I'm not saying these are necessary bad goalsat all, it's just that there's a huge disconnect between reality and the above model on which this change seems based on, and not a single mention about these goals and changes needed to get there. I mean, I totally get that you can't change everything at once, but if there's a plan this big behind something then maybe it should be brought up front, no?
- Panu -
So although rpm manages files under /var and /etc, it is not to the same extent as /usr. In this light it makes a lot of sense to hold the rpmdb state under /usr/ somewhere. The exact subdirectory doesn't really matter, it's just a matter of convention, so obviously we want to follow what opensuse and others are already using.
On Wed, Jan 12, 2022 at 11:24:49AM +0200, Panu Matilainen wrote:
On 1/12/22 11:05, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Jan 10, 2022 at 02:53:52PM -0700, Chris Murphy wrote:
Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt?
We have three similar locations: /usr (system vendor tree), /usr/local (admin non-packages installations), /opt (external vendor tree). In other words, both /usr and /opt are for packages, but from different sources. As an admin, you'd want to treat both package types the same, and e.g. roll them back together. So having a separate tree for /opt doesn't make much sense.
[At some point in the future] /opt should be renamed to /usr/opt and symlinked for backwards compat.
If /usr is to be truly portable and have e.g. 'rpm query, verify, remove, reinstall' work as expected, you need the metadata (the database) representing its state to always come along for the ride. Either the database is already in /usr, or you have to make sure /usr and /state are inseparable.
If /usr and /state are inseparable, and if rpm can also describe anything in /etc or /var or /opt, then all or part of those directories are also inseparable from /state. And thus /usr. So I think /state doesn't help.
Yeah, /state doesn't help with anything. It'd be just some part of the whole system state, without a clear separation of why this particular subset. My preference: /usr/lib/sysimage/rpm > /var/lib/rpm > /state.
To what degree do rpm and dnf intend to touch locations outside of /usr *and care* about tracking those changes?
Traditionally, packages installed all kinds of files all over the place. But we're slowly and painfully moving towards the model where:
- packages are only allowed to install under /usr, /var, and /etc. (Or under /opt, but I'd want to move that to /usr/opt…)
- packages must support /var/cache being wiped at any time, and most packages support anything under /var being wiped at any time.
- systemd and other projects are trying to only use /etc for local admin state, and support "factory reset" by wiping /etc and /var.
Oh, right. More hidden agenda behind this thing.
Panu,
I hope you realize that I'm not connected to this change in any shape or fashion and I'm just commenting on the published proposal like the other participants in this thread.
Zbyszek
When looking at it with these glasses on, it explains quite a few things about the change proposal, such as completely ignoring the fact that nearly all packages put something in /etc.
I'm not saying these are necessary bad goalsat all, it's just that there's a huge disconnect between reality and the above model on which this change seems based on, and not a single mention about these goals and changes needed to get there. I mean, I totally get that you can't change everything at once, but if there's a plan this big behind something then maybe it should be brought up front, no?
- Panu -
So although rpm manages files under /var and /etc, it is not to the same extent as /usr. In this light it makes a lot of sense to hold the rpmdb state under /usr/ somewhere. The exact subdirectory doesn't really matter, it's just a matter of convention, so obviously we want to follow what opensuse and others are already using.
On 1/12/22 11:53, Zbigniew Jędrzejewski-Szmek wrote:
On Wed, Jan 12, 2022 at 11:24:49AM +0200, Panu Matilainen wrote:
On 1/12/22 11:05, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Jan 10, 2022 at 02:53:52PM -0700, Chris Murphy wrote:
Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt?
We have three similar locations: /usr (system vendor tree), /usr/local (admin non-packages installations), /opt (external vendor tree). In other words, both /usr and /opt are for packages, but from different sources. As an admin, you'd want to treat both package types the same, and e.g. roll them back together. So having a separate tree for /opt doesn't make much sense.
[At some point in the future] /opt should be renamed to /usr/opt and symlinked for backwards compat.
If /usr is to be truly portable and have e.g. 'rpm query, verify, remove, reinstall' work as expected, you need the metadata (the database) representing its state to always come along for the ride. Either the database is already in /usr, or you have to make sure /usr and /state are inseparable.
If /usr and /state are inseparable, and if rpm can also describe anything in /etc or /var or /opt, then all or part of those directories are also inseparable from /state. And thus /usr. So I think /state doesn't help.
Yeah, /state doesn't help with anything. It'd be just some part of the whole system state, without a clear separation of why this particular subset. My preference: /usr/lib/sysimage/rpm > /var/lib/rpm > /state.
To what degree do rpm and dnf intend to touch locations outside of /usr *and care* about tracking those changes?
Traditionally, packages installed all kinds of files all over the place. But we're slowly and painfully moving towards the model where:
- packages are only allowed to install under /usr, /var, and /etc. (Or under /opt, but I'd want to move that to /usr/opt…)
- packages must support /var/cache being wiped at any time, and most packages support anything under /var being wiped at any time.
- systemd and other projects are trying to only use /etc for local admin state, and support "factory reset" by wiping /etc and /var.
Oh, right. More hidden agenda behind this thing.
Panu,
I hope you realize that I'm not connected to this change in any shape or fashion and I'm just commenting on the published proposal like the other participants in this thread.
Absolutely. I intended to thank you for bringing that behind-the-scenes grand plan up but seems it got lost in the edits. So thanks for bringing it up!
I actually do know the "factory reset" plan from way way back but had completely forgotten about it, and chances are I'm not alone in that. And I bet that not everybody in this discussion knew about it at all.
Others have complained about the lack of detail in the "benefit to Fedora" section, and I can only concur. At this point it's fairly obvious the change is just a tiny tiny part of a MUCH bigger agenda, and knowing and understanding the bigger agenda is necessary to make sense and valuate the proposal.
- Panu -
On Wed, Jan 12, 2022, at 4:24 AM, Panu Matilainen wrote:
Oh, right. More hidden agenda behind this thing. When looking at it with these glasses on, it explains quite a few things about the change proposal, such as completely ignoring the fact that nearly all packages put something in /etc.
Right. rpm-ostree uses ostree, which introduces /usr/etc which are the pristine default config files. /etc is 3-way merged by ostree. One of the major benefits of this that I really love is `ostree admin config-diff` - at any point we can show you machine-local changes from the default, and it's trivial to reset back to defaults without redownloading a whole RPM.
The semantics of /etc are also one of the things that is very different between ostree and other image based update systems, including the "systemd upstream vision" introduced a while back, which is basically that /etc starts empty, can be populated dynamically, but is otherwise not touched across system upgrades. The core problem with this is it introduces a new major hysteresis point which is your copy of /etc is mostly from the initial installed OS version. You don't get *new default* config files, and even worse, packages that drop config files still linger around in this model.
I personally think ostree's handling of /etc is one of the things that makes it feel much more like a Unix system than other image based update systems.
There's no hidden agenda - the goal is to support image based updates as well as client side snapshots, factory reset, etc. And we're shipping today versions of Fedora that do a lot of this, and we want to continue to improve it.
This is a bit of tangent, but if I am not mistaken, I can create RPMs to manage content of my home directory. How folks envision this would work in the context of this proposal? But I also wonder if this is something considered by e.g. systemd-homed.
I am asking this question, because move of the rpmdb into /usr is not conceptual (and I admit that even in the light of my question above, the /var is not really conceptual). So has anybody thought about more generic concept for rpmdb, which would cover the mountable directories?
Vít
Dne 12. 01. 22 v 10:05 Zbigniew Jędrzejewski-Szmek napsal(a):
On Mon, Jan 10, 2022 at 02:53:52PM -0700, Chris Murphy wrote:
Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt?
We have three similar locations: /usr (system vendor tree), /usr/local (admin non-packages installations), /opt (external vendor tree). In other words, both /usr and /opt are for packages, but from different sources. As an admin, you'd want to treat both package types the same, and e.g. roll them back together. So having a separate tree for /opt doesn't make much sense.
[At some point in the future] /opt should be renamed to /usr/opt and symlinked for backwards compat.
If /usr is to be truly portable and have e.g. 'rpm query, verify, remove, reinstall' work as expected, you need the metadata (the database) representing its state to always come along for the ride. Either the database is already in /usr, or you have to make sure /usr and /state are inseparable.
If /usr and /state are inseparable, and if rpm can also describe anything in /etc or /var or /opt, then all or part of those directories are also inseparable from /state. And thus /usr. So I think /state doesn't help.
Yeah, /state doesn't help with anything. It'd be just some part of the whole system state, without a clear separation of why this particular subset. My preference: /usr/lib/sysimage/rpm > /var/lib/rpm > /state.
To what degree do rpm and dnf intend to touch locations outside of /usr *and care* about tracking those changes?
Traditionally, packages installed all kinds of files all over the place. But we're slowly and painfully moving towards the model where:
- packages are only allowed to install under /usr, /var, and /etc. (Or under /opt, but I'd want to move that to /usr/opt…)
- packages must support /var/cache being wiped at any time, and most packages support anything under /var being wiped at any time.
- systemd and other projects are trying to only use /etc for local admin state, and support "factory reset" by wiping /etc and /var.
So although rpm manages files under /var and /etc, it is not to the same extent as /usr. In this light it makes a lot of sense to hold the rpmdb state under /usr/ somewhere. The exact subdirectory doesn't really matter, it's just a matter of convention, so obviously we want to follow what opensuse and others are already using.
Zbyszek _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Jan 12, 2022, at 4:05 AM, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Jan 10, 2022 at 02:53:52PM -0700, Chris Murphy wrote:
Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt?
We have three similar locations: /usr (system vendor tree), /usr/local (admin non-packages installations), /opt (external vendor tree). In other words, both /usr and /opt are for packages, but from different sources. As an admin, you'd want to treat both package types the same, and e.g. roll them back together. So having a separate tree for /opt doesn't make much sense.
[At some point in the future] /opt should be renamed to /usr/opt and symlinked for backwards compat.
Unfortunately, real world RPMs that install into /opt also have e.g. log files in /opt/somesoftware/log, not /var/log/somesoftware. So it can't be underneath the read-only /usr mount. This is why rpm-ostree just straight up rejects RPMs that install into /opt. https://github.com/coreos/rpm-ostree/issues/233
I think I agree with Chris that really what we want is a separate rpmdb for this. That would mean these packages don't participate in offline transactional updates, can't be rolled back etc.
On Wed, Jan 12, 2022 at 7:03 AM Colin Walters walters@verbum.org wrote:
On Wed, Jan 12, 2022, at 4:05 AM, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Jan 10, 2022 at 02:53:52PM -0700, Chris Murphy wrote:
Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt?
We have three similar locations: /usr (system vendor tree), /usr/local (admin non-packages installations), /opt (external vendor tree). In other words, both /usr and /opt are for packages, but from different sources. As an admin, you'd want to treat both package types the same, and e.g. roll them back together. So having a separate tree for /opt doesn't make much sense.
[At some point in the future] /opt should be renamed to /usr/opt and symlinked for backwards compat.
Unfortunately, real world RPMs that install into /opt also have e.g. log files in /opt/somesoftware/log, not /var/log/somesoftware. So it can't be underneath the read-only /usr mount. This is why rpm-ostree just straight up rejects RPMs that install into /opt. https://github.com/coreos/rpm-ostree/issues/233
The FHS says /opt is for static files. I think logs going in /opt is asking for trouble if you really care about your logs. I'm not super attached to enforcing /var/opt/<provider>/log versus dumping it in /var/log. I think I prefer the latter because it's consistent with /etc/opt/<provider>/ and is thus consistent. If you opt-in to /opt then you don't get to opt-out of all the opt's. (not the best pun)
I think I agree with Chris that really what we want is a separate rpmdb for this. That would mean these packages don't participate in offline transactional updates, can't be rolled back etc.
Maybe we could. If /opt is on a subvolume (of some kind) with its own rpmdb, it can be snapshot, update the *snapshot* rather than the currently active subvolume. Once update completes, snapshot it again - even could be read-only like /usr. Even could get signed with fsverity before it's made active. Oops the update broke things, let's just roll back /opt. But there are some nuances here I'm not sure about, like maybe you'd want to split up the updates such that you're not simultaneously updating /usr and /opt - maybe they should happen in separate transactions? *shrug*
I think (open)SUSE's proposed Btrfs subvolume layout for transactional updates is going to put /opt into /var - so /var/opt includes the static files.
On Wed, Jan 12, 2022 at 03:06:17PM -0700, Chris Murphy wrote:
Unfortunately, real world RPMs that install into /opt also have e.g. log files in /opt/somesoftware/log, not /var/log/somesoftware. So it can't be underneath the read-only /usr mount. This is why rpm-ostree just straight up rejects RPMs that install into /opt. https://github.com/coreos/rpm-ostree/issues/233
The FHS says /opt is for static files. I think logs going in /opt is asking for trouble if you really care about your logs. I'm not super attached to enforcing /var/opt/<provider>/log versus dumping it in /var/log. I think I prefer the latter because it's consistent with /etc/opt/<provider>/ and is thus consistent. If you opt-in to /opt then you don't get to opt-out of all the opt's. (not the best pun)
Let's please not recommend yet-another place to dump text logs. Services should log to the journal. This allows users to view the logs without hassle and provides log rotation and control over the total size of the logs. This is especially important for non-distro packages that install to /opt: they are particularly likely not to implement log rotation correctly if they try on their own.
Zbyszek
* Zbigniew Jędrzejewski-Szmek:
Traditionally, packages installed all kinds of files all over the place. But we're slowly and painfully moving towards the model where:
- packages are only allowed to install under /usr, /var, and /etc. (Or under /opt, but I'd want to move that to /usr/opt…)
- packages must support /var/cache being wiped at any time, and most packages support anything under /var being wiped at any time.
- systemd and other projects are trying to only use /etc for local admin state, and support "factory reset" by wiping /etc and /var.
Do you see RPM being used for distribution content only in the future?
Thanks, Florian
On Wed, Jan 12, 2022 at 04:20:31PM +0100, Florian Weimer wrote:
- Zbigniew Jędrzejewski-Szmek:
Traditionally, packages installed all kinds of files all over the place. But we're slowly and painfully moving towards the model where:
- packages are only allowed to install under /usr, /var, and /etc. (Or under /opt, but I'd want to move that to /usr/opt…)
- packages must support /var/cache being wiped at any time, and most packages support anything under /var being wiped at any time.
- systemd and other projects are trying to only use /etc for local admin state, and support "factory reset" by wiping /etc and /var.
Do you see RPM being used for distribution content only in the future?
No, not really. RPM is best for distribution content, but it's also good for independent-vendor and local packages.
Zbyszek
On Wed, Jan 12, 2022 at 2:07 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Mon, Jan 10, 2022 at 02:53:52PM -0700, Chris Murphy wrote:
Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt?
We have three similar locations: /usr (system vendor tree), /usr/local (admin non-packages installations), /opt (external vendor tree). In other words, both /usr and /opt are for packages, but from different sources. As an admin, you'd want to treat both package types the same, and e.g. roll them back together. So having a separate tree for /opt doesn't make much sense.
[At some point in the future] /opt should be renamed to /usr/opt and symlinked for backwards compat.
I'm not attached to it being moved one way or another. But if there's a need or want for /opt to be truly independent from /usr, the idea of rolling back /usr while keeping /opt current or vice versa, then I'd say the packages wanting to use /opt need to exclusively use /opt and variants like /etc/opt and /var/opt.
To what degree do rpm and dnf intend to touch locations outside of /usr *and care* about tracking those changes?
Traditionally, packages installed all kinds of files all over the place. But we're slowly and painfully moving towards the model where:
- packages are only allowed to install under /usr, /var, and /etc. (Or under /opt, but I'd want to move that to /usr/opt…)
- packages must support /var/cache being wiped at any time, and most packages support anything under /var being wiped at any time.
- systemd and other projects are trying to only use /etc for local admin state, and support "factory reset" by wiping /etc and /var.
Is /usr/etc something that could be a thing Fedora wide? The way rpm-ostree does it, it's making a copy of /etc and does a three way merge so you get updated config file defaults. But without that logic available, is it bonkers to consider applications learning to look to look first in /etc, and if the config isn't found there to go look in /usr/etc?
Is /usr/etc something that could be a thing Fedora wide? The way rpm-ostree does it, it's making a copy of /etc and does a three way merge so you get updated config file defaults. But without that logic available, is it bonkers to consider applications learning to look to look first in /etc, and if the config isn't found there to go look in /usr/etc?
A lot of packages have been following the systemd module wherein they check for /etc first and if it's empty they will use the version in /usr/lib[64]/<default_config_path>, but this is being done on an app-by-app basis. I think the OSTree solution is actually quite elegant, since it doesn't require every application to change its config file handling.
On Thu, Jan 13, 2022 at 6:07 AM Stephen Gallagher sgallagh@redhat.com wrote:
Is /usr/etc something that could be a thing Fedora wide? The way rpm-ostree does it, it's making a copy of /etc and does a three way merge so you get updated config file defaults. But without that logic available, is it bonkers to consider applications learning to look to look first in /etc, and if the config isn't found there to go look in /usr/etc?
A lot of packages have been following the systemd module wherein they check for /etc first and if it's empty they will use the version in /usr/lib[64]/<default_config_path>, but this is being done on an app-by-app basis. I think the OSTree solution is actually quite elegant, since it doesn't require every application to change its config file handling.
Is the OSTree solution, i.e. the three way merge, self-contained in rpm-ostree (on the client)? Would it be practical to split that out in a form that say, dnf or snapper or systemd could make use of?
On Thu, Jan 13, 2022 at 2:40 PM Chris Murphy lists@colorremedies.com wrote:
On Thu, Jan 13, 2022 at 6:07 AM Stephen Gallagher sgallagh@redhat.com wrote:
Is /usr/etc something that could be a thing Fedora wide? The way rpm-ostree does it, it's making a copy of /etc and does a three way merge so you get updated config file defaults. But without that logic available, is it bonkers to consider applications learning to look to look first in /etc, and if the config isn't found there to go look in /usr/etc?
A lot of packages have been following the systemd module wherein they check for /etc first and if it's empty they will use the version in /usr/lib[64]/<default_config_path>, but this is being done on an app-by-app basis. I think the OSTree solution is actually quite elegant, since it doesn't require every application to change its config file handling.
Is the OSTree solution, i.e. the three way merge, self-contained in rpm-ostree (on the client)? Would it be practical to split that out in a form that say, dnf or snapper or systemd could make use of?
Well, considering that a variation of this is built into dpkg (I took a wild guess that the concept originates from dpkg's config handling logic), I don't see a reason it couldn't eventually be built into RPM itself.
On Mon, Jan 10, 2022 at 02:53:52PM -0700, Chris Murphy wrote:
On Mon, Jan 10, 2022 at 9:20 AM David Cantrell dcantrell@redhat.com wrote:
On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
== Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE. [snip]
Moving the RPM database to /usr feels incorrect to me, but we should move it to gain the improvements as noted in the feature proposal.
Going back to the original discussions on moving rpmdb...
Preferred is a new top-level location in /usr, .e.g /usr/sysimage/rpm. Next is "least worst" in /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006764.html http://lists.rpm.org/pipermail/rpm-maint/2017-October/006722.html
And the convergence was on /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006785.html
I don't see how /state solves the problem, rather than just rearranging the chairs.
I've seen the rearranging the chairs comment in multiple locations now. If the location doesn't matter, why can't the RPM db stay in /var/lib/rpm?
Numerous followups have noted the requirement that /usr contain read-only content, that it be shareable across hosts, and similar concepts. While this may or may not doable now like we could in the past, the bigger thing to me is around the understanding of what /usr contains. It is generally understood that /usr contains [most of] the installed system. What I think is a bigger requirement or expection now is that one can tar up /usr and transport it to another system or virtual machine or container and expect that it will _probably_ work maybe with a bit of tinkering. This is a really valuable thing to have for developers. Moving the RPM database to this tree adds a component that is unnecessary and sort of out of place.
Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt?
If /usr is to be truly portable and have e.g. 'rpm query, verify, remove, reinstall' work as expected, you need the metadata (the database) representing its state to always come along for the ride. Either the database is already in /usr, or you have to make sure /usr and /state are inseparable.
If /usr and /state are inseparable, and if rpm can also describe anything in /etc or /var or /opt, then all or part of those directories are also inseparable from /state. And thus /usr. So I think /state doesn't help.
To what degree do rpm and dnf intend to touch locations outside of /usr *and care* about tracking those changes? I think rpm can't remain static for all time. It either needs to become aware of multiple root trees, and even mix and match top-level directories to create variable roots. And possibly even manage these things. Or it needs to constrain its reach to /usr and /opt. Whether /usr and /opt are tied together, or can be mix and match with their own rpmdb's, I have no strong opinion on.
These are valid points. My suggestion of /state (or another named top-level directory...I saw /meta suggested somewhere, maybe LWN...but I kind of don't want to imply references to Facebook) is on the basis of RPM operating with a single database on a given system. The idea of multiple RPM databases is interesting and I think there's value in discussing that, but that would be a separate item for discussion.
Right now, a system needs to support installation from the vendor, third party repos, and local system packages. All with a single RPM database. So I suggested /state to get that information out of /var but still keep it tied to the host in question.
The /opt tree introduces another wrinkle. It's one that's never really gained a lot of use in the Linux world, but where it has it's been treated as both a place where vendors can deliver software and a separate /usr/local. I think it's reasonable to assume vendor-delivered things in /opt can be thought of as more independent than what's in /usr, but that's just an assumption. My most recent views of things found in /opt from third party vendors shows that they are self-contained trees built with the mindset of "I don't trust what the system will provide me".
To move the RPM database in to /usr for the rpm-ostree snapshot and rollback case would also imply restrictions around what a user can and can't do with RPM on their system. If that's the case, then I would say rpm should stop being a general purpose tool for the user and become a backend only tool run by the OS. Let the users create new third party packaging systems to install things on their own system.
On Thu, Jan 13, 2022 at 8:49 AM David Cantrell dcantrell@redhat.com wrote:
On Mon, Jan 10, 2022 at 02:53:52PM -0700, Chris Murphy wrote:
I don't see how /state solves the problem, rather than just rearranging the chairs.
I've seen the rearranging the chairs comment in multiple locations now. If the location doesn't matter, why can't the RPM db stay in /var/lib/rpm?
It's all about consequences, not that it can't stay in /var/lib/rpm.
The rpmdb could go in its own subvolume "rpmdb" mounted at /var/lib/rpm, and snapshot it at the same time as subvolume "root" which contains /etc /usr /opt. Each snapshot is a revision. And a way to mount matching revisions is needed if there's more than one snapshot to rollback. e.g.
root-x86-64:fedora36.0 root-x86-64:fedora36.1 rpmdb-x86-64:fedora36.0 rpmdb-x86-64:fedora36.1
When booting root-x86-64:fedora36.1 the helper needs to 'mount -o subvol=rpmdb-x86-64:fedora36.1 --uuid=$fsuuid /var/lib/rpm'
When booting root-x86-64:fedora36.0, the helper needs to 'mount -o subvol=rpmdb-x86-64:fedora36.0 --uuid=$fsuuid /var/lib/rpm'
And for that matter, it could be state-x86-64:fedora36.0 and .1, mounted at /state.
The helper could rewrite /etc/fstab, or develop a "discoverable subvolumes" systemd generator that assembles things per a heuristic. https://lists.freedesktop.org/archives/systemd-devel/2021-November/047063.ht...
It's true that this change proposal doesn't go far enough. The issues with /var go beyond /var/lib/rpm.
What about /var/lib/selinux? It's owned by the selinux-policy-targeted package. Even though the files may not change often, it probably needs to be snapshot and rolled back with revision matching for /usr and rpmdb. It could be done with a subvolume or another file system with LVM. There might be 6 of these carve outs.
Instead of possibly 6+ file systems for LVM thin setups, there could be two: "system" is subject to snapshots and rollbacks, "variable" is not. The same "discoverable subvolumes" systemd-generator idea proposes working with plain directories using the same naming scheme, and would use bind mounts to just reassemble everything from those two file systems onto the FHS layout. Including read-only /usr.
This was brought up on devel@ list a year ago and discussed at length, explains the difficulties with the many carve outs approach (open)SUSE has been using, and the various pros and consequences to go about simplifying it. https://github.com/lnussel/lnussel.github.io/blob/fs/_posts/2020-12-16-fslay...
These are valid points. My suggestion of /state (or another named top-level directory...I saw /meta suggested somewhere, maybe LWN...but I kind of don't want to imply references to Facebook) is on the basis of RPM operating with a single database on a given system. The idea of multiple RPM databases is interesting and I think there's value in discussing that, but that would be a separate item for discussion.
Right now, a system needs to support installation from the vendor, third party repos, and local system packages. All with a single RPM database. So I suggested /state to get that information out of /var but still keep it tied to the host in question.
I don't think /state is a bad idea. Maybe it suffers the same issue as this change proposal, in that it doesn't go far enough?
The /opt tree introduces another wrinkle. It's one that's never really gained a lot of use in the Linux world, but where it has it's been treated as both a place where vendors can deliver software and a separate /usr/local. I think it's reasonable to assume vendor-delivered things in /opt can be thought of as more independent than what's in /usr, but that's just an assumption. My most recent views of things found in /opt from third party vendors shows that they are self-contained trees built with the mindset of "I don't trust what the system will provide me".
I guess the main issue here for snapshots and rollbacks is, would it be ok to put /opt and /usr on the same lifecycle? They get snapshot and rolled back together? Or should /opt be a 3rd party playground and perhaps not even be subject to snapshots and rollbacks, it's entirely managed by rpm only? I don't know the answer to that. But I kinda like the idea of separate /opt with its own rpmdb. That way 'rm -rf /opt/*' is a valid way to reset the 3rd party location without messing up rpmdb because it too is cleaned up.
To move the RPM database in to /usr for the rpm-ostree snapshot and rollback case would also imply restrictions around what a user can and can't do with RPM on their system. If that's the case, then I would say rpm should stop being a general purpose tool for the user and become a backend only tool run by the OS. Let the users create new third party packaging systems to install things on their own system.
I'm not sure about this.
-- Chris Murphy
On Fri, Jan 14, 2022, at 2:46 AM, Chris Murphy wrote:
What about /var/lib/selinux? It's owned by the selinux-policy-targeted package. Even though the files may not change often, it probably needs to be snapshot and rolled back with revision matching for /usr and rpmdb.
Yep, welcome to https://bugzilla.redhat.com/show_bug.cgi?id=1290659
Once upon a time, Ben Cotton bcotton@redhat.com said:
== Benefit to Fedora ==
- The RPM database primarily describes the state of `/usr`. Storing
the databases in `/usr` will more easily facilitate OS rollback, without affecting `/var`.
Rolling back to the start... this statement is only marginally true. And "primarily" is a hand-wavy hedge that just pretends the rest doesn't matter.
Ony my Fedora 35 desktop, I have things in RPM under a lot of top-level directories:
$ rpm -qla | grep '^/.*/' | cut -d/ -f2 | sort | uniq -c 99 boot 2998 etc 14792 lib 43 lib64 97 opt 5 root 43 run 10 sbin 442175 usr 2036 var
Some of those are RPMs that should probably be updated, but some of that won't go away. There's important stuff in /var and /etc, and even for the stuff that could be blown away and recreated, most programs aren't set up (or don't have permission) to recreate their directories.
You can't snapshot just /usr, make software changes, and then roll it back, because programs in /usr have expectations about other directories. The whole OS doesn't exclusively exist in /usr. What if that /usr change was an updated version of MariaDB or PostgresSQL that updated the DB file format, or even just a program using one of those DBs that had a schema update?
The only way I see any benefit to trying to rearrange the RPM database would be to split it up somehow, where it could be spread across filesystems (but that still has the same consistency issue of rolling back /usr without making the same rollback to /var and /etc).
On Sun, Jan 16, 2022 at 5:05 PM Chris Adams linux@cmadams.net wrote:
Once upon a time, Ben Cotton bcotton@redhat.com said:
== Benefit to Fedora ==
- The RPM database primarily describes the state of `/usr`. Storing
the databases in `/usr` will more easily facilitate OS rollback, without affecting `/var`.
Rolling back to the start... this statement is only marginally true. And "primarily" is a hand-wavy hedge that just pretends the rest doesn't matter.
Ony my Fedora 35 desktop, I have things in RPM under a lot of top-level directories:
$ rpm -qla | grep '^/.*/' | cut -d/ -f2 | sort | uniq -c 99 boot 2998 etc 14792 lib 43 lib64 97 opt 5 root 43 run 10 sbin 442175 usr 2036 var
Some of those are RPMs that should probably be updated, but some of that won't go away. There's important stuff in /var and /etc, and even for the stuff that could be blown away and recreated, most programs aren't set up (or don't have permission) to recreate their directories.
I'm not expecting to do filesystem rollbacks to revert Firefox or SQLite. I expect if I have some kind of boot or startup problem, the system is a candidate for a rollback. In such a failure, I haven't run any other programs yet that may have upgraded a database.
Could modification to something in /etc result in startup failure? Yep. Is a full system rollback really called for here? It'll work, but it's a hammer. So maybe services need to learn to be a little suspicious of /etc configs, and fail safe? Maybe system critical items should be able to fall back to /usr/etc?
-
Alexander Sosedkin -
Ben Cotton -
Chris Adams -
Chris Murphy -
Colin Walters -
David Cantrell -
Dominik 'Rathann' Mierzejewski -
Dominique Martinet -
Fabio Valentini -
Florian Weimer -
Francisco J. Tsao Santín -
Frank Ch. Eigler -
Gordon Messmer -
Lennart Poettering -
Matthew Miller -
Michael Catanzaro -
Michel Alexandre Salim -
Neal Gompa -
Nico Kadel-Garcia -
Panu Matilainen -
Peter Boy -
Peter Robinson -
Richard W.M. Jones -
Sebastian Crane -
Stephen Gallagher -
Stephen John Smoogen -
Steve Grubb -
Sérgio Basto -
Tom Hughes -
Tomasz Kłoczko -
Vitaly Zaitsev -
Vít Ondruch -
Zbigniew Jędrzejewski-Szmek