5:49 a.m.
https://bugzilla.redhat.com/show_bug.cgi?id=2042147#c2
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
https://bugzilla.redhat.com/2042099
The RPM database is supposed to move from /var/lib/rpm to
/usr/lib/sysimage/rpm. This was supposed to happen automatically when
you upgraded the rpm package from an earlier version to 4.17.0-10.fc36
(Feb-Mar 2022).
On several machines it is reported that the migration was only half-
completed. The symptoms are that the RPM database is still in
/var/lib/rpm (/usr contains symlinks to it). See typical output from
failed & successful migrations at the end of the email.
So _if_ you have rpm >= 4.17.0-10.fc36 installed:
- Do you see the symptom of a failed migration? Or does it appear
to be successful? (Or neither case??)
- Did you:
* Install F37 or Rawhide from scratch?
* Upgrade using ordinary dnf update or similar?
* Upgrade using dnf system-upgrade?
* Some other install/upgrade method?
Rich.
** After a failed migration: **
$ ls -la /var/lib/rpm
total 339004
drwxr-xr-x. 2 root root 4096 Feb 16 2022 .
drwxr-xr-x. 76 root root 4096 Aug 23 13:28 ..
-rw-r--r--. 1 root root 0 Oct 18 14:28 .migratedb
-rw-r--r--. 1 root root 347095040 Oct 26 16:50 rpmdb.sqlite
-rw-r--r--. 1 root root 32768 Oct 27 15:28 rpmdb.sqlite-shm
-rw-r--r--. 1 root root 0 Oct 26 16:50 rpmdb.sqlite-wal
-rw-r--r--. 1 root root 0 Feb 16 2022 .rpm.lock
$ ls -la /usr/lib/sysimage/rpm
total 8
drwxr-xr-x. 2 root root 4096 Oct 5 12:05 .
drwxr-xr-x. 3 root root 4096 Feb 9 2022 ..
lrwxrwxrwx. 1 root root 34 Oct 18 14:28 .migratedb ->
../../../../var/lib/rpm/.migratedb
lrwxrwxrwx. 1 root root 36 Oct 18 14:28 rpmdb.sqlite ->
../../../../var/lib/rpm/rpmdb.sqlite
lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-shm ->
../../../../var/lib/rpm/rpmdb.sqlite-shm
lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-wal ->
../../../../var/lib/rpm/rpmdb.sqlite-wal
lrwxrwxrwx. 1 root root 33 Oct 18 14:28 .rpm.lock ->
../../../../var/lib/rpm/.rpm.lock
** After a successful migration: **
$ ls -la /var/lib/rpm
lrwxrwxrwx. 1 root root 26 Sep 5 13:35 /var/lib/rpm -> ../../usr/lib/sysimage/rpm
$ ls -la /usr/lib/sysimage/rpm
total 316324
drwxr-xr-x. 2 root root 91 Sep 17 23:03 .
drwxr-xr-x. 3 root root 17 Aug 9 14:27 ..
-rw-r--r--. 1 root root 323878912 Oct 27 16:12 rpmdb.sqlite
-rw-r--r--. 1 root root 32768 Oct 28 10:45 rpmdb.sqlite-shm
-rw-r--r--. 1 root root 0 Oct 27 16:12 rpmdb.sqlite-wal
-rw-r--r--. 1 root root 0 Sep 5 13:53 .rpm.lock
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
6:24 a.m.
I have one machine that has failed.
It was an upgrade from 35 to 36 done using dnf distro-sync
but I have plenty of others done the same way that worked.
One difference is that it's a machine that wasn't upgraded
until August while other ones were done back in May as a
result of which the upgrade was:
from rpm-4.17.1-3.fc35.x86_64
to rpm-4.17.1-3.fc36.x86_64
while other machines were more like:
from rpm-4.17.0-4.fc35.x86_64
to rpm-4.17.0-10.fc36.x86_64
Tom
On 28/10/2022 11:49, Richard W.M. Jones wrote:
https://bugzilla.redhat.com/show_bug.cgi?id=2042147#c2
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
https://bugzilla.redhat.com/2042099
The RPM database is supposed to move from /var/lib/rpm to
/usr/lib/sysimage/rpm. This was supposed to happen automatically when
you upgraded the rpm package from an earlier version to 4.17.0-10.fc36
(Feb-Mar 2022).
On several machines it is reported that the migration was only half-
completed. The symptoms are that the RPM database is still in
/var/lib/rpm (/usr contains symlinks to it). See typical output from
failed & successful migrations at the end of the email.
So _if_ you have rpm >= 4.17.0-10.fc36 installed:
- Do you see the symptom of a failed migration? Or does it appear
to be successful? (Or neither case??)
- Did you:
* Install F37 or Rawhide from scratch?
* Upgrade using ordinary dnf update or similar?
* Upgrade using dnf system-upgrade?
* Some other install/upgrade method?
Rich.
** After a failed migration: **
$ ls -la /var/lib/rpm
total 339004
drwxr-xr-x. 2 root root 4096 Feb 16 2022 .
drwxr-xr-x. 76 root root 4096 Aug 23 13:28 ..
-rw-r--r--. 1 root root 0 Oct 18 14:28 .migratedb
-rw-r--r--. 1 root root 347095040 Oct 26 16:50 rpmdb.sqlite
-rw-r--r--. 1 root root 32768 Oct 27 15:28 rpmdb.sqlite-shm
-rw-r--r--. 1 root root 0 Oct 26 16:50 rpmdb.sqlite-wal
-rw-r--r--. 1 root root 0 Feb 16 2022 .rpm.lock
$ ls -la /usr/lib/sysimage/rpm
total 8
drwxr-xr-x. 2 root root 4096 Oct 5 12:05 .
drwxr-xr-x. 3 root root 4096 Feb 9 2022 ..
lrwxrwxrwx. 1 root root 34 Oct 18 14:28 .migratedb ->
../../../../var/lib/rpm/.migratedb
lrwxrwxrwx. 1 root root 36 Oct 18 14:28 rpmdb.sqlite ->
../../../../var/lib/rpm/rpmdb.sqlite
lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-shm ->
../../../../var/lib/rpm/rpmdb.sqlite-shm
lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-wal ->
../../../../var/lib/rpm/rpmdb.sqlite-wal
lrwxrwxrwx. 1 root root 33 Oct 18 14:28 .rpm.lock ->
../../../../var/lib/rpm/.rpm.lock
** After a successful migration: **
$ ls -la /var/lib/rpm
lrwxrwxrwx. 1 root root 26 Sep 5 13:35 /var/lib/rpm -> ../../usr/lib/sysimage/rpm
$ ls -la /usr/lib/sysimage/rpm
total 316324
drwxr-xr-x. 2 root root 91 Sep 17 23:03 .
drwxr-xr-x. 3 root root 17 Aug 9 14:27 ..
-rw-r--r--. 1 root root 323878912 Oct 27 16:12 rpmdb.sqlite
-rw-r--r--. 1 root root 32768 Oct 28 10:45 rpmdb.sqlite-shm
-rw-r--r--. 1 root root 0 Oct 27 16:12 rpmdb.sqlite-wal
-rw-r--r--. 1 root root 0 Sep 5 13:53 .rpm.lock
--
Tom Hughes (tom(a)compton.nu)
http://compton.nu/
6:29 a.m.
The reason it hadn't completed is that rpmdb-migrate.service
was enabled on that machine.
Enabling (and starting) that service made it complete.
Tom
On 28/10/2022 12:24, Tom Hughes via devel wrote:
I have one machine that has failed.
It was an upgrade from 35 to 36 done using dnf distro-sync
but I have plenty of others done the same way that worked.
One difference is that it's a machine that wasn't upgraded
until August while other ones were done back in May as a
result of which the upgrade was:
from rpm-4.17.1-3.fc35.x86_64
to rpm-4.17.1-3.fc36.x86_64
while other machines were more like:
from rpm-4.17.0-4.fc35.x86_64
to rpm-4.17.0-10.fc36.x86_64
Tom
On 28/10/2022 11:49, Richard W.M. Jones wrote:
> https://bugzilla.redhat.com/show_bug.cgi?id=2042147#c2
> https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
> https://bugzilla.redhat.com/2042099
>
> The RPM database is supposed to move from /var/lib/rpm to
> /usr/lib/sysimage/rpm. This was supposed to happen automatically when
> you upgraded the rpm package from an earlier version to 4.17.0-10.fc36
> (Feb-Mar 2022).
>
> On several machines it is reported that the migration was only half-
> completed. The symptoms are that the RPM database is still in
> /var/lib/rpm (/usr contains symlinks to it). See typical output from
> failed & successful migrations at the end of the email.
>
> So _if_ you have rpm >= 4.17.0-10.fc36 installed:
>
> - Do you see the symptom of a failed migration? Or does it appear
> to be successful? (Or neither case??)
>
> - Did you:
> * Install F37 or Rawhide from scratch?
> * Upgrade using ordinary dnf update or similar?
> * Upgrade using dnf system-upgrade?
> * Some other install/upgrade method?
>
> Rich.
>
>
> ** After a failed migration: **
>
> $ ls -la /var/lib/rpm
> total 339004
> drwxr-xr-x. 2 root root 4096 Feb 16 2022 .
> drwxr-xr-x. 76 root root 4096 Aug 23 13:28 ..
> -rw-r--r--. 1 root root 0 Oct 18 14:28 .migratedb
> -rw-r--r--. 1 root root 347095040 Oct 26 16:50 rpmdb.sqlite
> -rw-r--r--. 1 root root 32768 Oct 27 15:28 rpmdb.sqlite-shm
> -rw-r--r--. 1 root root 0 Oct 26 16:50 rpmdb.sqlite-wal
> -rw-r--r--. 1 root root 0 Feb 16 2022 .rpm.lock
>
> $ ls -la /usr/lib/sysimage/rpm
> total 8
> drwxr-xr-x. 2 root root 4096 Oct 5 12:05 .
> drwxr-xr-x. 3 root root 4096 Feb 9 2022 ..
> lrwxrwxrwx. 1 root root 34 Oct 18 14:28 .migratedb ->
> ../../../../var/lib/rpm/.migratedb
> lrwxrwxrwx. 1 root root 36 Oct 18 14:28 rpmdb.sqlite ->
> ../../../../var/lib/rpm/rpmdb.sqlite
> lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-shm ->
> ../../../../var/lib/rpm/rpmdb.sqlite-shm
> lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-wal ->
> ../../../../var/lib/rpm/rpmdb.sqlite-wal
> lrwxrwxrwx. 1 root root 33 Oct 18 14:28 .rpm.lock ->
> ../../../../var/lib/rpm/.rpm.lock
>
>
> ** After a successful migration: **
>
> $ ls -la /var/lib/rpm
> lrwxrwxrwx. 1 root root 26 Sep 5 13:35 /var/lib/rpm ->
> ../../usr/lib/sysimage/rpm
>
> $ ls -la /usr/lib/sysimage/rpm
> total 316324
> drwxr-xr-x. 2 root root 91 Sep 17 23:03 .
> drwxr-xr-x. 3 root root 17 Aug 9 14:27 ..
> -rw-r--r--. 1 root root 323878912 Oct 27 16:12 rpmdb.sqlite
> -rw-r--r--. 1 root root 32768 Oct 28 10:45 rpmdb.sqlite-shm
> -rw-r--r--. 1 root root 0 Oct 27 16:12 rpmdb.sqlite-wal
> -rw-r--r--. 1 root root 0 Sep 5 13:53 .rpm.lock
>
>
--
Tom Hughes (tom(a)compton.nu)
http://compton.nu/
7:31 a.m.
On Fri, Oct 28, 2022 at 12:29:30PM +0100, Tom Hughes wrote:
The reason it hadn't completed is that rpmdb-migrate.service
was enabled on that machine.
[was not, I guess?]
Enabling (and starting) that service made it complete.
Interesting. The current state of that service is:
○ rpmdb-migrate.service - RPM database migration to /usr
Loaded: loaded (/usr/lib/systemd/system/rpmdb-migrate.service; disabled; vendor
preset: enabled)
Active: inactive (dead)
There are no log entries for this service, but my logs only go back to
around April which is probably too late to see anything.
After starting the service:
Oct 28 13:29:33 systemd[1]: Starting rpmdb-migrate.service - RPM database migration to
/usr...
Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/.migratedb'
Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/rpmdb.sqlite-shm'
Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/rpmdb.sqlite'
Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/rpmdb.sqlite-wal'
Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/.rpm.lock'
Oct 28 13:29:35 rpmdb_migrate[1722092]: removed directory '/var/lib/rpm'
Oct 28 13:29:35 rpmdb_migrate[1722468]: '/var/lib/rpm' ->
'../../usr/lib/sysimage/rpm'
Oct 28 13:29:35 systemd[1]: rpmdb-migrate.service: Deactivated successfully.
Oct 28 13:29:35 systemd[1]: Finished rpmdb-migrate.service - RPM database migration to
/usr.
Oct 28 13:29:35 systemd[1]: rpmdb-migrate.service: Consumed 2.433s CPU time.
... and the migration has been successful. So at least we know how to
fix this. Although it's quite curious that the service was installed
and supposed to run but didn't.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
7:45 a.m.
On 28/10/2022 13:31, Richard W.M. Jones wrote:
On Fri, Oct 28, 2022 at 12:29:30PM +0100, Tom Hughes wrote:
> The reason it hadn't completed is that rpmdb-migrate.service
> was enabled on that machine.
[was not, I guess?]
Yes ;-)
> Enabling (and starting) that service made it complete.
Interesting. The current state of that service is:
○ rpmdb-migrate.service - RPM database migration to /usr
Loaded: loaded (/usr/lib/systemd/system/rpmdb-migrate.service; disabled; vendor
preset: enabled)
Active: inactive (dead)
There are no log entries for this service, but my logs only go back to
around April which is probably too late to see anything.
After starting the service:
Oct 28 13:29:33 systemd[1]: Starting rpmdb-migrate.service - RPM database migration to
/usr...
Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/.migratedb'
Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/rpmdb.sqlite-shm'
Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/rpmdb.sqlite'
Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/rpmdb.sqlite-wal'
Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/.rpm.lock'
Oct 28 13:29:35 rpmdb_migrate[1722092]: removed directory '/var/lib/rpm'
Oct 28 13:29:35 rpmdb_migrate[1722468]: '/var/lib/rpm' ->
'../../usr/lib/sysimage/rpm'
Oct 28 13:29:35 systemd[1]: rpmdb-migrate.service: Deactivated successfully.
Oct 28 13:29:35 systemd[1]: Finished rpmdb-migrate.service - RPM database migration to
/usr.
Oct 28 13:29:35 systemd[1]: rpmdb-migrate.service: Consumed 2.433s CPU time.
... and the migration has been successful. So at least we know how to
fix this. Although it's quite curious that the service was installed
and supposed to run but didn't.
The idea is that the service is enabled (not sure off hand what is
supposed to do that) but not started, so that it runs when you reboot
and completes the migration as part of the boot.
When it runs it removes /var/lib/rpm/.migratedb which was created
by the rpm scripts and hence prevents the service running on future
boots as it's no longer needed.
Tom
--
Tom Hughes (tom(a)compton.nu)
http://compton.nu/
7:49 a.m.
On Fri, Oct 28, 2022 at 01:45:32PM +0100, Tom Hughes wrote:
On 28/10/2022 13:31, Richard W.M. Jones wrote:
>On Fri, Oct 28, 2022 at 12:29:30PM +0100, Tom Hughes wrote:
>>The reason it hadn't completed is that rpmdb-migrate.service
>>was enabled on that machine.
>[was not, I guess?]
Yes ;-)
>>Enabling (and starting) that service made it complete.
>
>Interesting. The current state of that service is:
>
>○ rpmdb-migrate.service - RPM database migration to /usr
> Loaded: loaded (/usr/lib/systemd/system/rpmdb-migrate.service; disabled; vendor
preset: enabled)
> Active: inactive (dead)
>
>There are no log entries for this service, but my logs only go back to
>around April which is probably too late to see anything.
>
>After starting the service:
>
>Oct 28 13:29:33 systemd[1]: Starting rpmdb-migrate.service - RPM database migration
to /usr...
>Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/.migratedb'
>Oct 28 13:29:35 rpmdb_migrate[1722092]: removed
'/var/lib/rpm/rpmdb.sqlite-shm'
>Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/rpmdb.sqlite'
>Oct 28 13:29:35 rpmdb_migrate[1722092]: removed
'/var/lib/rpm/rpmdb.sqlite-wal'
>Oct 28 13:29:35 rpmdb_migrate[1722092]: removed '/var/lib/rpm/.rpm.lock'
>Oct 28 13:29:35 rpmdb_migrate[1722092]: removed directory '/var/lib/rpm'
>Oct 28 13:29:35 rpmdb_migrate[1722468]: '/var/lib/rpm' ->
'../../usr/lib/sysimage/rpm'
>Oct 28 13:29:35 systemd[1]: rpmdb-migrate.service: Deactivated successfully.
>Oct 28 13:29:35 systemd[1]: Finished rpmdb-migrate.service - RPM database migration
to /usr.
>Oct 28 13:29:35 systemd[1]: rpmdb-migrate.service: Consumed 2.433s CPU time.
>
>... and the migration has been successful. So at least we know how to
>fix this. Although it's quite curious that the service was installed
>and supposed to run but didn't.
The idea is that the service is enabled (not sure off hand what is
supposed to do that) but not started, so that it runs when you reboot
and completes the migration as part of the boot.
When it runs it removes /var/lib/rpm/.migratedb which was created
by the rpm scripts and hence prevents the service running on future
boots as it's no longer needed.
It's hard to tell what happened without logs going back all the way,
but the machine was rebooted 65 days ago and several times before
that. Within the available logs there is no mention of the service so
it may already have tried to run on a boot prior to the logs and
failed for some reason.
We do have reports of this happening from 3 independent people now so
it's not a one-off.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
6:40 a.m.
I've just checked 4 systems (2x Workstation, MATE and Cloud), and the
migration was successful on all of them. They were all upgraded to the
latest versions in testing at the time with dnf system-upgrade.
Regular package updates were performed weekly at the latest.
8:08 a.m.
Hi,
I upgraded from 35 to 36 in the beginning of Sept this year. The migration failed.
May I help somehow?
Regards,
Sergey.
9:46 a.m.
On Fri, 28 Oct 2022 13:08:19 -0000
"Sergey Mende" <sergey(a)mende.spb.ru> wrote:
I upgraded from 35 to 36 in the beginning of Sept this year. The
migration failed. May I help somehow?
I used the advice from Tom Hughes in his earlier message to complete my
failed migration.
# systemctl enable rpmdb-migrate
# systemctl start rpmdb-migrate
When that completed, the migration was successfully done.
I am running F37, started as F35 rawhide, updated using dnf,
usually daily, with reboots also usually daily. I am currently running
rpm-4.18.0-1.fc37.x86_64.
9:50 a.m.
On Fri, Oct 28, 2022 at 07:46:03AM -0700, stan via devel wrote:
On Fri, 28 Oct 2022 13:08:19 -0000
"Sergey Mende" <sergey(a)mende.spb.ru> wrote:
> I upgraded from 35 to 36 in the beginning of Sept this year. The
> migration failed. May I help somehow?
I used the advice from Tom Hughes in his earlier message to complete my
failed migration.
# systemctl enable rpmdb-migrate
# systemctl start rpmdb-migrate
When that completed, the migration was successfully done.
I am running F37, started as F35 rawhide, updated using dnf,
usually daily, with reboots also usually daily. I am currently running
rpm-4.18.0-1.fc37.x86_64.
Correct -- while we're trying to find out how widespread the problem
is, the way to fix it on individual machines is as outlined above.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
9:59 a.m.
Rich, stan,
thank you, I got the recipe from the previous conversations. I just asked if I could help
with logs analysis or somehow else before I fix the issue.
Regards,
Sergey
11:53 p.m.
Hi,
Richard W.M. Jones wrote:
https://bugzilla.redhat.com/show_bug.cgi?id=2042147#c2
https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
https://bugzilla.redhat.com/2042099
The RPM database is supposed to move from /var/lib/rpm to
/usr/lib/sysimage/rpm. This was supposed to happen automatically when
you upgraded the rpm package from an earlier version to 4.17.0-10.fc36
(Feb-Mar 2022).
On several machines it is reported that the migration was only half-
completed. The symptoms are that the RPM database is still in
/var/lib/rpm (/usr contains symlinks to it). See typical output from
failed & successful migrations at the end of the email.
So _if_ you have rpm >= 4.17.0-10.fc36 installed:
- Do you see the symptom of a failed migration? Or does it appear
to be successful? (Or neither case??)
- Did you:
* Install F37 or Rawhide from scratch?
* Upgrade using ordinary dnf update or similar?
* Upgrade using dnf system-upgrade?
* Some other install/upgrade method?
I upgraded two systems from f35 to f36 in the past week via
dnf system-upgrade. Neither of them completed the rpm
migration.
I _think_ the issue is due to the version-release in the
%triggerun which should enable the rpmdb-migrate service.
That is:
%triggerun -- rpm < 4.17.0-7
# Handle rpmdb migrate service on erasure of old to avoid ordering issues
if [ -x /usr/bin/systemctl ]; then
systemctl --no-reload preset rpmdb-migrate ||:
fi
That was accurate when it was added in 0b9f813 (Migrate
rpmdb to /usr/lib/sysimage/rpm (#2042099), 2022-01-26).
However, when rpm was updated in f35 in e9927df (Rebase to
rpm 4.17.1 (http://rpm.org/wiki/Releases/4.17.1),
2022-07-01), which never had the migration code, it
prevented any up-to-date f35 system from triggering the
migration.
Anyone upgrading from f35 after rpm-4.17.1 was pushed to
f35 won't have the rpmdb-migrate service enabled and the
database will remain in /var/lib/rpm (with .migratedb) and
symlinks in /usr/lib/sysimage/rpm.
It seems that it's not so much a failed migration as a
migration which is never attempted. :/
--
Todd
9:49 a.m.
On 10/28/22 22:53, Todd Zullinger wrote:
Hi,
Richard W.M. Jones wrote:
> https://bugzilla.redhat.com/show_bug.cgi?id=2042147#c2
> https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
> https://bugzilla.redhat.com/2042099
>
> The RPM database is supposed to move from /var/lib/rpm to
> /usr/lib/sysimage/rpm. This was supposed to happen automatically when
> you upgraded the rpm package from an earlier version to 4.17.0-10.fc36
> (Feb-Mar 2022).
>
> On several machines it is reported that the migration was only half-
> completed. The symptoms are that the RPM database is still in
> /var/lib/rpm (/usr contains symlinks to it). See typical output from
> failed & successful migrations at the end of the email.
>
> So _if_ you have rpm >= 4.17.0-10.fc36 installed:
>
> - Do you see the symptom of a failed migration? Or does it appear
> to be successful? (Or neither case??)
>
> - Did you:
> * Install F37 or Rawhide from scratch?
> * Upgrade using ordinary dnf update or similar?
> * Upgrade using dnf system-upgrade?
> * Some other install/upgrade method?
I upgraded two systems from f35 to f36 in the past week via
dnf system-upgrade. Neither of them completed the rpm
migration.
I _think_ the issue is due to the version-release in the
%triggerun which should enable the rpmdb-migrate service.
That is:
%triggerun -- rpm < 4.17.0-7
# Handle rpmdb migrate service on erasure of old to avoid ordering issues
if [ -x /usr/bin/systemctl ]; then
systemctl --no-reload preset rpmdb-migrate ||:
fi
That was accurate when it was added in 0b9f813 (Migrate
rpmdb to /usr/lib/sysimage/rpm (#2042099), 2022-01-26).
However, when rpm was updated in f35 in e9927df (Rebase to
rpm 4.17.1 (http://rpm.org/wiki/Releases/4.17.1),
2022-07-01), which never had the migration code, it
prevented any up-to-date f35 system from triggering the
migration.
Anyone upgrading from f35 after rpm-4.17.1 was pushed to
f35 won't have the rpmdb-migrate service enabled and the
database will remain in /var/lib/rpm (with .migratedb) and
symlinks in /usr/lib/sysimage/rpm.
It seems that it's not so much a failed migration as a
migration which is never attempted. :/
Likely in certain situations.
What I don't understand is why all the trickery around enabling the
service - why not just enable it unconditionally? It already has a
condition:
ConditionPathExists=/var/lib/rpm/.migratedb
and so won't run multiple times.
In my case though it seems to be a product of a bad conditional. I'm
running a continually updating rawhide VM and:
Jan 06 10:16:09 vmrawhide-rufous.cora.nwra.com systemd[1]:
rpmdb-rebuild.service - RPM database rebuild was skipped because of a
failed condition check (ConditionPathExists=/var/lib/rpm/.rebuilddb).
...
Feb 06 19:35:14 vmrawhide-rufous.cora.nwra.com systemd[1]:
rpmdb-rebuild.service - RPM database rebuild was skipped because of a
failed condition check (ConditionPathExists=/var/lib/rpm/.rebuilddb).
Feb 08 19:56:31 vmrawhide-rufous.cora.nwra.com systemd[1]:
rpmdb-rebuild.service - RPM database rebuild was skipped because of a
failed condition check
(ConditionPathExists=/usr/lib/sysimage/rpm/.rebuilddb).
...
Oct 16 09:47:01 vmrawhide-rufous.cora.nwra.com systemd[1]:
rpmdb-rebuild.service - RPM database rebuild was skipped because of a
failed condition check
(ConditionPathExists=/usr/lib/sysimage/rpm/.rebuilddb).
Oct 27 17:32:57 vmrawhide-rufous.cora.nwra.com systemd[1]:
rpmdb-rebuild.service - RPM database rebuild was skipped because of an
unmet condition check
(ConditionPathExists=/usr/lib/sysimage/rpm/.rebuilddb).
# systemctl status rpmdb-rebuild.service
○ rpmdb-rebuild.service - RPM database rebuild
Loaded: loaded (/usr/lib/systemd/system/rpmdb-rebuild.service;
enabled; preset: enabled)
Active: inactive (dead)
Condition: start condition failed at Thu 2022-10-27 17:32:57 MDT; 1
day 15h ago
Oct 27 17:32:57 vmrawhide-rufous.cora.nwra.com systemd[1]:
rpmdb-rebuild.service - RPM database rebuild was skipped because of an
unmet condition check
(ConditionPathExists=/usr/lib/sysimage/rpm/.rebuilddb).
# ls -la /usr/lib/sysimage/rpm/
total 8
drwxr-xr-x. 2 root root 4096 Oct 5 05:05 .
drwxr-xr-x. 3 root root 4096 Aug 9 07:27 ..
lrwxrwxrwx. 1 root root 34 Oct 5 20:32 .migratedb ->
../../../../var/lib/rpm/.migratedb
lrwxrwxrwx. 1 root root 36 Oct 5 20:32 rpmdb.sqlite ->
../../../../var/lib/rpm/rpmdb.sqlite
lrwxrwxrwx. 1 root root 40 Oct 5 20:32 rpmdb.sqlite-shm ->
../../../../var/lib/rpm/rpmdb.sqlite-shm
lrwxrwxrwx. 1 root root 40 Oct 5 20:32 rpmdb.sqlite-wal ->
../../../../var/lib/rpm/rpmdb.sqlite-wal
lrwxrwxrwx. 1 root root 33 Oct 5 20:32 .rpm.lock ->
../../../../var/lib/rpm/.rpm.lock
# ls -la /var/lib/rpm
total 444064
drwxr-xr-x. 2 root root 4096 Feb 6 2022 .
drwxr-xr-x. 68 root root 4096 Aug 9 07:27 ..
-rw-r--r--. 1 root root 0 Oct 5 20:35 .migratedb
-rw-r--r--. 1 root root 454676480 Oct 28 20:27 rpmdb.sqlite
-rw-r--r--. 1 root root 32768 Oct 29 08:09 rpmdb.sqlite-shm
-rw-r--r--. 1 root root 0 Oct 28 20:27 rpmdb.sqlite-wal
-rw-r--r--. 1 root root 0 Feb 6 2022 .rpm.lock
--
Orion Poplawski
he/him/his - surely the least important thing about me
IT Systems Manager 720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 https://www.nwra.com/
10:37 a.m.
On Sat, Oct 29, 2022 at 10:49 AM Orion Poplawski <orion(a)nwra.com> wrote:
On 10/28/22 22:53, Todd Zullinger wrote:
> Hi,
>
> Richard W.M. Jones wrote:
>> https://bugzilla.redhat.com/show_bug.cgi?id=2042147#c2
>> https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr
>> https://bugzilla.redhat.com/2042099
>>
>> The RPM database is supposed to move from /var/lib/rpm to
>> /usr/lib/sysimage/rpm. This was supposed to happen automatically when
>> you upgraded the rpm package from an earlier version to 4.17.0-10.fc36
>> (Feb-Mar 2022).
>>
>> On several machines it is reported that the migration was only half-
>> completed. The symptoms are that the RPM database is still in
>> /var/lib/rpm (/usr contains symlinks to it). See typical output from
>> failed & successful migrations at the end of the email.
>>
>> So _if_ you have rpm >= 4.17.0-10.fc36 installed:
>>
>> - Do you see the symptom of a failed migration? Or does it appear
>> to be successful? (Or neither case??)
>>
>> - Did you:
>> * Install F37 or Rawhide from scratch?
>> * Upgrade using ordinary dnf update or similar?
>> * Upgrade using dnf system-upgrade?
>> * Some other install/upgrade method?
>
> I upgraded two systems from f35 to f36 in the past week via
> dnf system-upgrade. Neither of them completed the rpm
> migration.
>
> I _think_ the issue is due to the version-release in the
> %triggerun which should enable the rpmdb-migrate service.
> That is:
>
> %triggerun -- rpm < 4.17.0-7
> # Handle rpmdb migrate service on erasure of old to avoid ordering issues
> if [ -x /usr/bin/systemctl ]; then
> systemctl --no-reload preset rpmdb-migrate ||:
> fi
>
> That was accurate when it was added in 0b9f813 (Migrate
> rpmdb to /usr/lib/sysimage/rpm (#2042099), 2022-01-26).
>
> However, when rpm was updated in f35 in e9927df (Rebase to
> rpm 4.17.1 (http://rpm.org/wiki/Releases/4.17.1),
> 2022-07-01), which never had the migration code, it
> prevented any up-to-date f35 system from triggering the
> migration.
>
> Anyone upgrading from f35 after rpm-4.17.1 was pushed to
> f35 won't have the rpmdb-migrate service enabled and the
> database will remain in /var/lib/rpm (with .migratedb) and
> symlinks in /usr/lib/sysimage/rpm.
>
> It seems that it's not so much a failed migration as a
> migration which is never attempted. :/
Likely in certain situations.
What I don't understand is why all the trickery around enabling the
service - why not just enable it unconditionally? It already has a
condition:
I was being careful because I wasn't initially sure what approach I'd
use. It changed a couple of times. It can probably change now to be
unconditional activation during install/upgrade of rpm, though.
--
真実はいつも一つ！/ Always, there's only one truth!
11:23 a.m.
On 29-10-2022 17:37, Neal Gompa wrote:
On Sat, Oct 29, 2022 at 10:49 AM Orion Poplawski
<orion(a)nwra.com> wrote:
>
> On 10/28/22 22:53, Todd Zullinger wrote:
<snip>
>> It seems that it's not so much a failed migration as a
>> migration which is never attempted. :/
>
> Likely in certain situations.
>
> What I don't understand is why all the trickery around enabling the
> service - why not just enable it unconditionally? It already has a
> condition:
>
I was being careful because I wasn't initially sure what approach I'd
use. It changed a couple of times. It can probably change now to be
unconditional activation during install/upgrade of rpm, though.
Now may be a good time to get this fixed. Once F37 is released and again
once F35 goes EOL a large number of users will upgrade.
-- Sandro
538
days inactive
539
days old
15 comments
9 participants
participants (9)
-
Alexander Ploumistos -
Neal Gompa -
Orion Poplawski -
Richard W.M. Jones -
Sandro -
Sergey Mende -
stan -
Todd Zullinger -
Tom Hughes