yum updates corrupting rpmdb

[Joe Smith]

This is the second update in as many days that has resulted in a 
corrupted rpm database for me. I was able to fix yesterday's with a 
--rebuilddb and 'yum clean all' but before I go through that little 
exercise again, is anyone else seeing problems? Any suggestions how to 
debug this?

# yum update
( lists a dozen or so packages to update, including firefox...
   ...I say 'y'...
   ...10 minutes later the terminal history is filled with:
rpmdb: PANIC: fatal region error detected; run recovery
rpmdb: PANIC: fatal region error detected; run recovery

and yum won't die except by 'kill -9'

# rpm -qa
rpmdb: PANIC: fatal region error detected; run recovery
error: db4 error(-30977) from dbenv->open: DB_RUNRECOVERY: Fatal error, 
run database recovery
error: cannot open Packages index using db3 -  (-30977)
error: cannot open Packages database in /var/lib/rpm

Not out of space...
# df /var/lib/rpm
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sdb7               497829    205804    266323  44% /var

and there are no kernel disk error messages.

Is there some "run recovery" that's different than --rebuilddb?

<Joe

maybe yum-updatesd? (was Re: yum updates corrupting rpmdb)

[Joe Smith]

Joe Smith wrote:
>  ...
>  I realize this is purely circumstantial, and I have no explanation why 
>  no one else is reporting trouble, if there's a problem with yum-updatesd.
>  
>  Since it still wouldn't prove anything to re-enable yum-updatesd and see 
>  the problem return (or not), and I don't care to fiddle with rebuilddb 
>  without a chance to get some solid evidence in return, I'm just going to 
>  leave yum-updatesd off, for now.
>  ... 
And just now I had the database go belly up even with yum-updatesd 
completely disabled, so it looks like that's off the hook.

I wonder how the same hardware was stable all through f7, and f8 up 
until about a month ago. Maybe a kernel glitch with my hardware? The 
system is stable otherwise, but maybe the database exercises something 
unusual.

Still scratching my head...

<Joe

maybe yum-updatesd? (was Re: yum updates corrupting rpmdb)

[Kam Leo]

On Jan 3, 2008 9:15 PM, Joe Smith <jes(a)martnet.com&gt; wrote:
>  Joe Smith wrote:
>  > ...
>  > I realize this is purely circumstantial, and I have no explanation why
>  > no one else is reporting trouble, if there's a problem with yum-updatesd.
>  >
>  > Since it still wouldn't prove anything to re-enable yum-updatesd and see
>  > the problem return (or not), and I don't care to fiddle with rebuilddb
>  > without a chance to get some solid evidence in return, I'm just going to
>  > leave yum-updatesd off, for now.
>  > ...
>
>  And just now I had the database go belly up even with yum-updatesd
>  completely disabled, so it looks like that's off the hook.
>
>  I wonder how the same hardware was stable all through f7, and f8 up
>  until about a month ago. Maybe a kernel glitch with my hardware? The
>  system is stable otherwise, but maybe the database exercises something
>  unusual.
>
>  Still scratching my head...
>
>
>  <Joe 
There's a chance that your hardware (power supply, fan, RAM, hard
drive, etc.) is failing and you are observing the symptoms.

[Craig White]

On Sat, 2007-12-15 at 23:54 -0500, Joe Smith wrote:
>  Craig White wrote:
>  > ...
>  >> Is there some "run recovery" that's different than --rebuilddb?
>  > ----
>  > no but perhaps your /dev/sdb7 has some corruption itself.
>  > 
>  > You might try something like 'shutdown now -Fr' to force an fsck on
>  > reboot
>  
>  Thanks for the suggestion, but no, it checks clean. Badblocks comes up 
>  clean also.
>  
>  I tried dropping to single user to run fsck, but couldn't unmount /var. 
>  There were no open files on it, so maybe it's connected to whatever was 
>  preventing yum from being killed; maybe a database block was locked or 
>  something, I don't know.
>  
>  It could be last time, because I rebuilt the database before I rebooted, 
>  there was still this strange problem with the partition, so maybe the 
>  database still was not clean.
>  
>  So, I rebooted to single-user; rpm --rebuilddb; yum clean all
>  
>  Now rpm is working again, but yum reports no updates are available, even 
>  though there are packages announced for updates that should apply to me:
>  
>  # yum list firefox gnome-python2, ...
>  Installed Packages
>  autofs.i386          1:5.0.2-17             installed
>  firefox.i386         2.0.0.10-2.fc8         installed
>  gnome-python2.i386   2.20.0-1.fc8           installed
>  lftp.i386            3.5.14-2.fc8           installed
>  libshout.i386        2.2.2-1.fc6            installed
>  libsoup.i386         2.2.103-1.fc8          installed
>  nspluginwrapper.i386 0.9.91.5-12.fc8        installed
>  system-config-firewall.noarch  1.0.11-1.fc8 installed
>  
>  Yet, todays update annoucements list:
>  
>  [SECURITY] Fedora 8 Update: autofs-5.0.2-20  updates
>  Fedora 8 Update: firefox-2.0.0.10-3.fc8  updates
>  Fedora 8 Update: gnome-python2-2.20.1-1.fc8  updates
>  Fedora 8 Update: lftp-3.5.14-3.fc8  updates
>  Fedora 8 Update: libshout-2.2.2-2.fc8  updates
>  Fedora 8 Update: libsoup-2.2.104-1.fc8  updates
>  Fedora 8 Update: nspluginwrapper-0.9.91.5-13.fc8  updates
>  Fedora 8 Update: system-config-firewall-1.0.12-2.fc8  updates
>  
>  So it looks like all these packages should be upgraded, but yum says no.
>  
>  I remember at least a few of those being in the update list before yum 
>  went berserk, so something sure looks off-track here.
>  
>  What could cause that? Maybe the updates have not reached the mirrors 
>  yet? Surely yum doesn't depend on that to determine what upgrades are 
>  available.
>  
>  Hmm... out of desperation, I ran another "yum clean all" and now it's 
>  picking up all those updates. So I wasn't crazy--yum was just fouled up 
>  again.
>  
>  Suggestions welcome--let's see if I can actually do the updates this 
>  time. Two down, five left. 
----
did you want some cheese with that whine?

man yum.conf

metadata_expire
Time  (in  seconds) after which the metadata will expire. So that if the
current metadata downloaded is less than this many seconds old then yum
will not update the metadata against the repository.  If you find that
yum is not downloading information on updates as often as you would like
lower the value of this option.

Craig

maybe yum-updatesd? (was Re: yum updates corrupting rpmdb)

[Craig White]

On Fri, 2008-01-04 at 12:17 -0500, Joe Smith wrote:
>  Craig White wrote:
>  > ...
>  > almost sounds like an incomplete upgrade occurred, perhaps with the
>  > wrong version of berkley-db4 or python installed.
>  > 
>  > Did you ever go through things like installing yum-tools and
>  > package-cleanup?
>  
>  Hey, thanks for the ideas!
>  
>  "rpm -V db4" was clean--something else I should check?
>  
>  I always assumed that if "rpm --rebuilddb" succeeded, then the database 
>  should be ok. But it looks like my assumption was wrong:
>  
>  # cd /var/lib/rpm/
>  # /usr/lib/rpm/rpmdb_verify Packages
>  # /usr/lib/rpm/rpmdb_verify Requirename
>  db_verify: Page 223: bad prev_pgno 0 on overflow page (should be 119)
>  db_verify: Page 223: overflow item incomplete
>  db_verify: Requirename: DB_VERIFY_BAD: Database verification failed
>  # for f in *; do echo "$f"; /usr/lib/rpm/rpmdb_verify $f; done
>  Basenames
>  Conflictname
>  Dirnames
>  db_verify: Page 1982: bad page number 191
>  db_verify: Dirnames: DB_VERIFY_BAD: Database verification failed
>  Filemd5s
>  db_verify: Page 4635: bad page number 89
>  db_verify: Filemd5s: DB_VERIFY_BAD: Database verification failed
>  Group
>  db_verify: Page 18: overflow page of invalid type 2
>  db_verify: Group: DB_VERIFY_BAD: Database verification failed
>  Installtid
>  Name
>  Packages
>  Providename
>  db_verify: Page 342: bad page number 105
>  db_verify: Providename: DB_VERIFY_BAD: Database verification failed
>  Provideversion
>  db_verify: Page 110: bad prev_pgno 0 on overflow page (should be 112)
>  db_verify: Page 110: overflow item incomplete
>  db_verify: Provideversion: DB_VERIFY_BAD: Database verification failed
>  Pubkeys
>  Requirename
>  db_verify: Page 223: bad prev_pgno 0 on overflow page (should be 119)
>  db_verify: Page 223: overflow item incomplete
>  db_verify: Requirename: DB_VERIFY_BAD: Database verification failed
>  Requireversion
>  db_verify: Page 191: partially zeroed page
>  db_verify: Requireversion: DB_VERIFY_BAD: Database verification failed
>  Sha1header
>  Sigmd5
>  db_verify: Page 32: bad page number 223
>  db_verify: Page 32: invalid prev_pgno 119
>  db_verify: Sigmd5: DB_VERIFY_BAD: Database verification failed
>  Triggername
>  
>  I found an old (2003) suggestion for rebuilding the Packages db:
>   > mv Packages Packages-ORIG
>   > /usr/lib/rpm/rpmdb_dump Packages-ORIG |
>   >   /usr/lib/rpm/rpmdb_load Packages
>  
>  Running that on each db...
>  # for f in Requireversion Provideversion Providename Group Filemd5s 
>  Dirnames; do echo "$f"; mv "$f" "$f.orig" &&
> /usr/lib/rpm/rpmdb_dump 
>  "$f.orig" | /usr/lib/rpm/rpmdb_load "$f"; done
>  Requireversion
>  db_load: Transactional Data Store incompatible with environment
>  Provideversion
>  db_load: Transactional Data Store incompatible with environment
>  Providename
>  db_load: Transactional Data Store incompatible with environment
>  Group
>  db_load: Transactional Data Store incompatible with environment
>  Filemd5s
>  db_load: Transactional Data Store incompatible with environment
>  Dirnames
>  db_load: Transactional Data Store incompatible with environment
>  
>  I have no idea what that message means, but it doesn't mean the command 
>  failed, apparently:
>  
>  # for f in `ls | grep -v '\.'`; do echo "$f"; /usr/lib/rpm/rpmdb_verify
>
>  $f; done
>  Basenames
>  Conflictname
>  Dirnames
>  Filemd5s
>  Group
>  Installtid
>  Name
>  Packages
>  Providename
>  Provideversion
>  Pubkeys
>  Requirename
>  Requireversion
>  Sha1header
>  Sigmd5
>  Triggername
>  
>  Looks clean now.
>  
>  I'm running a "rpm -Va" just now, and so far it looks reasonable, so 
>  maybe the problem is done now.
>  
>  I have no idea what I'm doing here, and I still don't understand if 
>  "--rebuilddb" was missing something, or what.
>  
>  Clues welcomed.
>  
>  PS: Oops. No, now the "rpm -Va" is finding stuff that looks like some 
>  parts of the last update that failed were partially installed. Maybe I 
>  need to go back and repeat the latest package installs to make sure 
>  everything is clean.
>  
>  Maybe time to punt and re-install f8. 
----
see this...

http://fedoraproject.org/wiki/YumUpgradeFaq

see point #4 'Tips' about yum-utils/package-cleanup

Craig

maybe yum-updatesd? (was Re: yum updates corrupting rpmdb)

[Joe Smith]

Joe Smith wrote:
>  ...
>  I'm running a "rpm -Va" just now, and so far it looks reasonable, so 
>  maybe the problem is done now.
>  ... 
Nope. Yum can even successfully finish an update which apparently leaves 
errors in the database:

# yum update
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package gnupg2.i386 0:2.0.8-1.fc8 set to be updated
---> Package gdb.i386 0:6.6-39.fc8 set to be updated
---> Package liferea.i386 0:1.4.10-1.fc8 set to be updated
---> Package libEMF.i386 0:1.0.3-5.fc8 set to be updated
---> Package yum-utils.noarch 0:1.1.9-2.fc8 set to be updated
---> Package libcdio.i386 0:0.78.2-4.fc8 set to be updated
--> Finished Dependency Resolution
   ...
Updated: gdb.i386 0:6.6-39.fc8 gnupg2.i386 0:2.0.8-1.fc8 libEMF.i386 
0:1.0.3-5.fc8 libcdio.i386 0:0.78.2-4.fc8 liferea.i386 0:1.4.10-1.fc8 
yum-utils.noarch 0:1.1.9-2.fc8
Complete!
# cd /var/lib/rpm
# for f in `ls | grep -v '\.'`; do /usr/lib/rpm/rpmdb_verify $f; done
db_verify: Page 2355: partially zeroed page
db_verify: Dirnames: DB_VERIFY_BAD: Database verification failed
db_verify: Page 641: bad page number 247
db_verify: Providename: DB_VERIFY_BAD: Database verification failed
db_verify: Page 109: partially zeroed page
db_verify: Provideversion: DB_VERIFY_BAD: Database verification failed
db_verify: Page 247: partially zeroed page
db_verify: Requireversion: DB_VERIFY_BAD: Database verification failed
db_verify: Page 160: bad page number 109
db_verify: Sha1header: DB_VERIFY_BAD: Database verification failed

If it hadn't ever worked, I'd be looking at hardware, but it's worked 
well for many months (f7/8), and I'm seeing no drive errors or 
instability otherwise.

<Joe

maybe yum-updatesd? (was Re: yum updates corrupting rpmdb)

[Joe Smith]

Craig White wrote:
>  ...
>  almost sounds like an incomplete upgrade occurred, perhaps with the
>  wrong version of berkley-db4 or python installed.
>  
>  Did you ever go through things like installing yum-tools and
>  package-cleanup? 
Hey, thanks for the ideas!

"rpm -V db4" was clean--something else I should check?

I always assumed that if "rpm --rebuilddb" succeeded, then the database 
should be ok. But it looks like my assumption was wrong:

# cd /var/lib/rpm/
# /usr/lib/rpm/rpmdb_verify Packages
# /usr/lib/rpm/rpmdb_verify Requirename
db_verify: Page 223: bad prev_pgno 0 on overflow page (should be 119)
db_verify: Page 223: overflow item incomplete
db_verify: Requirename: DB_VERIFY_BAD: Database verification failed
# for f in *; do echo "$f"; /usr/lib/rpm/rpmdb_verify $f; done
Basenames
Conflictname
Dirnames
db_verify: Page 1982: bad page number 191
db_verify: Dirnames: DB_VERIFY_BAD: Database verification failed
Filemd5s
db_verify: Page 4635: bad page number 89
db_verify: Filemd5s: DB_VERIFY_BAD: Database verification failed
Group
db_verify: Page 18: overflow page of invalid type 2
db_verify: Group: DB_VERIFY_BAD: Database verification failed
Installtid
Name
Packages
Providename
db_verify: Page 342: bad page number 105
db_verify: Providename: DB_VERIFY_BAD: Database verification failed
Provideversion
db_verify: Page 110: bad prev_pgno 0 on overflow page (should be 112)
db_verify: Page 110: overflow item incomplete
db_verify: Provideversion: DB_VERIFY_BAD: Database verification failed
Pubkeys
Requirename
db_verify: Page 223: bad prev_pgno 0 on overflow page (should be 119)
db_verify: Page 223: overflow item incomplete
db_verify: Requirename: DB_VERIFY_BAD: Database verification failed
Requireversion
db_verify: Page 191: partially zeroed page
db_verify: Requireversion: DB_VERIFY_BAD: Database verification failed
Sha1header
Sigmd5
db_verify: Page 32: bad page number 223
db_verify: Page 32: invalid prev_pgno 119
db_verify: Sigmd5: DB_VERIFY_BAD: Database verification failed
Triggername

I found an old (2003) suggestion for rebuilding the Packages db:
>  mv Packages Packages-ORIG
>  /usr/lib/rpm/rpmdb_dump Packages-ORIG |
>    /usr/lib/rpm/rpmdb_load Packages 
Running that on each db...
# for f in Requireversion Provideversion Providename Group Filemd5s 
Dirnames; do echo "$f"; mv "$f" "$f.orig" &&
/usr/lib/rpm/rpmdb_dump 
"$f.orig" | /usr/lib/rpm/rpmdb_load "$f"; done
Requireversion
db_load: Transactional Data Store incompatible with environment
Provideversion
db_load: Transactional Data Store incompatible with environment
Providename
db_load: Transactional Data Store incompatible with environment
Group
db_load: Transactional Data Store incompatible with environment
Filemd5s
db_load: Transactional Data Store incompatible with environment
Dirnames
db_load: Transactional Data Store incompatible with environment

I have no idea what that message means, but it doesn't mean the command 
failed, apparently:

# for f in `ls | grep -v '\.'`; do echo "$f"; /usr/lib/rpm/rpmdb_verify

$f; done
Basenames
Conflictname
Dirnames
Filemd5s
Group
Installtid
Name
Packages
Providename
Provideversion
Pubkeys
Requirename
Requireversion
Sha1header
Sigmd5
Triggername

Looks clean now.

I'm running a "rpm -Va" just now, and so far it looks reasonable, so 
maybe the problem is done now.

I have no idea what I'm doing here, and I still don't understand if 
"--rebuilddb" was missing something, or what.

Clues welcomed.

PS: Oops. No, now the "rpm -Va" is finding stuff that looks like some 
parts of the last update that failed were partially installed. Maybe I 
need to go back and repeat the latest package installs to make sure 
everything is clean.

Maybe time to punt and re-install f8.

<Joe

maybe yum-updatesd? (was Re: yum updates corrupting rpmdb)

[Craig White]

On Fri, 2008-01-04 at 00:15 -0500, Joe Smith wrote:
>  Joe Smith wrote:
>  > ...
>  > I realize this is purely circumstantial, and I have no explanation why 
>  > no one else is reporting trouble, if there's a problem with yum-updatesd.
>  > 
>  > Since it still wouldn't prove anything to re-enable yum-updatesd and see 
>  > the problem return (or not), and I don't care to fiddle with rebuilddb 
>  > without a chance to get some solid evidence in return, I'm just going to 
>  > leave yum-updatesd off, for now.
>  > ...
>  
>  And just now I had the database go belly up even with yum-updatesd 
>  completely disabled, so it looks like that's off the hook.
>  
>  I wonder how the same hardware was stable all through f7, and f8 up 
>  until about a month ago. Maybe a kernel glitch with my hardware? The 
>  system is stable otherwise, but maybe the database exercises something 
>  unusual. 
----
almost sounds like an incomplete upgrade occurred, perhaps with the
wrong version of berkley-db4 or python installed.

Did you ever go through things like installing yum-tools and
package-cleanup?

Craig

maybe yum-updatesd? (was Re: yum updates corrupting rpmdb)

[Joe Smith]

Craig White wrote:
>  ... 
>  Last time I suggested that you peruse/search bugzilla for other reports
>  as probably the best way to see if there are reports similar to yours. 
Ack! You did, and I forgot to look.

So... lots of nasty "rpmdb: PANIC: fatal region error detected..." 
reports, mostly fc6 or earlier.

There was one nasty kernel mmap bug, but that was against fc6 and fixed 
for kernels >= 2.6.9, although it seems at least one person is still 
experiencing it with f7/f8:
https://bugzilla.redhat.com/show_bug.cgi?id=181363

Anyway, it seems that the rpmdb exercises the system in some unusual 
ways, and it's not unheard of to see "mysterious" corruption that can 
depend on what hardware is present.

I'm just going to stick with manual update checks for now.

<Joe

maybe yum-updatesd? (was Re: yum updates corrupting rpmdb)

[Craig White]

On Sat, 2007-12-29 at 22:00 -0500, Joe Smith wrote:
>  Joe Smith wrote:
>  > This is the second update in as many days that has resulted in a 
>  > corrupted rpm database for me. ...
>  
>  Just an update, FWIW.
>  
>  I continued to have routine rpmdb corruption issues, sometimes with as 
>  simple a command as "yum remove foo-1.2.3-1.fc8". The rebuilddb 
>  procedure was always successful and allowed me to proceed with minimal 
>  package management, but after the next reboot (it's just a desktop: 
>  booted mornings; shut down at night), yum commands would encounter 
>  corruption again.
>  
>  This continued up until about a week ago, when I decided that I would 
>  try shutting off yum-updatesd altogether, and since then I've had zero 
>  problems with any yum commands, including two updates involving several 
>  packages.
>  
>  I realize this is purely circumstantial, and I have no explanation why 
>  no one else is reporting trouble, if there's a problem with yum-updatesd.
>  
>  Since it still wouldn't prove anything to re-enable yum-updatesd and see 
>  the problem return (or not), and I don't care to fiddle with rebuilddb 
>  without a chance to get some solid evidence in return, I'm just going to 
>  leave yum-updatesd off, for now.
>  
>  If anyone has any thoughts or suggestions for further steps to 
>  understanding this, I'd be most interested to hear them. 
----
I have seen little to no complaints regarding this other than yours. I
suspect that it is not an endemic problem (ala RHL 8.0)

Last time I suggested that you peruse/search bugzilla for other reports
as probably the best way to see if there are reports similar to yours.

Craig

maybe yum-updatesd? (was Re: yum updates corrupting rpmdb)

[Joe Smith]

Joe Smith wrote:
>  This is the second update in as many days that has resulted in a 
>  corrupted rpm database for me. ... 
Just an update, FWIW.

I continued to have routine rpmdb corruption issues, sometimes with as 
simple a command as "yum remove foo-1.2.3-1.fc8". The rebuilddb 
procedure was always successful and allowed me to proceed with minimal 
package management, but after the next reboot (it's just a desktop: 
booted mornings; shut down at night), yum commands would encounter 
corruption again.

This continued up until about a week ago, when I decided that I would 
try shutting off yum-updatesd altogether, and since then I've had zero 
problems with any yum commands, including two updates involving several 
packages.

I realize this is purely circumstantial, and I have no explanation why 
no one else is reporting trouble, if there's a problem with yum-updatesd.

Since it still wouldn't prove anything to re-enable yum-updatesd and see 
the problem return (or not), and I don't care to fiddle with rebuilddb 
without a chance to get some solid evidence in return, I'm just going to 
leave yum-updatesd off, for now.

If anyone has any thoughts or suggestions for further steps to 
understanding this, I'd be most interested to hear them.

<Joe

[Joe Smith]

Todd Denniston wrote:
>  ...
>  try
>  df -h / /var /usr
>  
>  I don't remember what all went wrong when it happened to me (been a few 
>  years), but if the system does not have enough space to hold the update 
>  rpms AND apply the updates at the same time, then you can get several 
>  things about doing updates messed up.
>  
>  On systems with limited disk space, I usually hand yum update a few 
>  packages at a time with `yum clean`s in between each yum update.
>  
>  at 200MB free in /var/ it looks like you have enough space to download 
>  all of the updates, the question is: are / and /usr similarly 
>  provisioned with free space? 
Good thought, thanks.

I have exactly the same problem, that's why I checked the free space. I 
just didn't show all the partitions ;-)

/, /tmp and /usr were all <100% as well. But it's possible that space 
was freed up before I checked it.

I've run out of space during an update lots of times and never had a 
database problem. Either a download fails and the update stops, or an 
install fails (space check) and it stops--always cleanly in the past.

The total amount ofd space for this batch (~30M) was within my 
seat-of-the-pants "safe" zone.

But, you're right, I could easily see how running out of space at just 
the wrong time could cause the db to corrupt, and I can't rule out the 
possibility that could've happened this time.

So far, so good. I just have to wait and see what happens when the next 
updates land.

Thanks for your help.

<Joe

[Todd Denniston]

Joe Smith wrote, On 12/15/2007 11:54 PM:
>  Craig White wrote:
> > ...
> >> Is there some "run recovery" that's different than --rebuilddb?
> > ----
> > no but perhaps your /dev/sdb7 has some corruption itself.
> >
> > You might try something like 'shutdown now -Fr' to force an fsck on
> > reboot
>  
>  Thanks for the suggestion, but no, it checks clean. Badblocks comes up 
>  clean also.
>   
<SNIP>
>  
>  So it looks like all these packages should be upgraded, but yum says no.
>  
>  I remember at least a few of those being in the update list before yum 
>  went berserk, so something sure looks off-track here.
>   
<SNIP>

>  Hmm... out of desperation, I ran another "yum clean all"
> and now it's 
>  picking up all those updates. So I wasn't crazy--yum was just fouled up 
>  again.
>  
>  Suggestions welcome--let's see if I can actually do the updates this 
>  time. Two down, five left.
>  
>  <Joe
>   
try
df -h / /var /usr

I don't remember what all went wrong when it happened to me (been a few 
years), but if the system does not have enough space to hold the update rpms 
AND apply the updates at the same time, then you can get several things about 
doing updates messed up.

On systems with limited disk space, I usually hand yum update a few packages 
at a time with `yum clean`s in between each yum update.

at 200MB free in /var/ it looks like you have enough space to download all of 
the updates, the question is: are / and /usr similarly provisioned with free 
space?

-- 
Todd Denniston
Crane Division, Naval Surface Warfare Center (NSWC Crane)
Harnessing the Power of Technology for the Warfighter

[Joe Smith]

Craig White wrote:
>  ...
> > Is there some "run recovery" that's different than --rebuilddb?
>  ----
>  no but perhaps your /dev/sdb7 has some corruption itself.
>  
>  You might try something like 'shutdown now -Fr' to force an fsck on
>  reboot 
Thanks for the suggestion, but no, it checks clean. Badblocks comes up 
clean also.

I tried dropping to single user to run fsck, but couldn't unmount /var. 
There were no open files on it, so maybe it's connected to whatever was 
preventing yum from being killed; maybe a database block was locked or 
something, I don't know.

It could be last time, because I rebuilt the database before I rebooted, 
there was still this strange problem with the partition, so maybe the 
database still was not clean.

So, I rebooted to single-user; rpm --rebuilddb; yum clean all

Now rpm is working again, but yum reports no updates are available, even 
though there are packages announced for updates that should apply to me:

# yum list firefox gnome-python2, ...
Installed Packages
autofs.i386          1:5.0.2-17             installed
firefox.i386         2.0.0.10-2.fc8         installed
gnome-python2.i386   2.20.0-1.fc8           installed
lftp.i386            3.5.14-2.fc8           installed
libshout.i386        2.2.2-1.fc6            installed
libsoup.i386         2.2.103-1.fc8          installed
nspluginwrapper.i386 0.9.91.5-12.fc8        installed
system-config-firewall.noarch  1.0.11-1.fc8 installed

Yet, todays update annoucements list:

[SECURITY] Fedora 8 Update: autofs-5.0.2-20  updates
Fedora 8 Update: firefox-2.0.0.10-3.fc8  updates
Fedora 8 Update: gnome-python2-2.20.1-1.fc8  updates
Fedora 8 Update: lftp-3.5.14-3.fc8  updates
Fedora 8 Update: libshout-2.2.2-2.fc8  updates
Fedora 8 Update: libsoup-2.2.104-1.fc8  updates
Fedora 8 Update: nspluginwrapper-0.9.91.5-13.fc8  updates
Fedora 8 Update: system-config-firewall-1.0.12-2.fc8  updates

So it looks like all these packages should be upgraded, but yum says no.

I remember at least a few of those being in the update list before yum 
went berserk, so something sure looks off-track here.

What could cause that? Maybe the updates have not reached the mirrors 
yet? Surely yum doesn't depend on that to determine what upgrades are 
available.

Hmm... out of desperation, I ran another "yum clean all" and now it's 
picking up all those updates. So I wasn't crazy--yum was just fouled up 
again.

Suggestions welcome--let's see if I can actually do the updates this 
time. Two down, five left.

<Joe

[Craig White]

On Sat, 2007-12-15 at 19:52 -0500, Joe Smith wrote:
>  This is the second update in as many days that has resulted in a 
>  corrupted rpm database for me. I was able to fix yesterday's with a 
>  --rebuilddb and 'yum clean all' but before I go through that little 
>  exercise again, is anyone else seeing problems? Any suggestions how to 
>  debug this?
>  
>  # yum update
>  ( lists a dozen or so packages to update, including firefox...
>     ...I say 'y'...
>     ...10 minutes later the terminal history is filled with:
>  rpmdb: PANIC: fatal region error detected; run recovery
>  rpmdb: PANIC: fatal region error detected; run recovery
>  
>  and yum won't die except by 'kill -9'
>  
>  # rpm -qa
>  rpmdb: PANIC: fatal region error detected; run recovery
>  error: db4 error(-30977) from dbenv->open: DB_RUNRECOVERY: Fatal error, 
>  run database recovery
>  error: cannot open Packages index using db3 -  (-30977)
>  error: cannot open Packages database in /var/lib/rpm
>  
>  Not out of space...
>  # df /var/lib/rpm
>  Filesystem           1K-blocks      Used Available Use% Mounted on
>  /dev/sdb7               497829    205804    266323  44% /var
>  
>  and there are no kernel disk error messages.
>  
>  Is there some "run recovery" that's different than --rebuilddb? 
----
no but perhaps your /dev/sdb7 has some corruption itself.

You might try something like 'shutdown now -Fr' to force an fsck on
reboot

Craig