rpmdb and /var filesystem corruption in F7+

[Orion Poplawski]

I have a lot of problems with rpmdb corruption occurring during
(network) installs since around February 2007, and with yum updates 
afterwards.  I've reported the problem here 
<https://bugzilla.redhat.com/show_bug.cgi?id=230362>;.

The problem seems to have gotten worse with the latest rawhide releases
- I think about one of every five attempts to perform an install 
succeeds.  I'm still somewhat baffled that I appear to be the only 
person to be reporting these problems considering how reproducible they 
are for me with all kinds of hardware.

I keep a separate /var partition, and recently one of our F7 systems' 
/var filesystem got corrupted 
<https://bugzilla.redhat.com/show_bug.cgi?id=305301>;.  Perhaps the 
problems are related.

Anyways, I'm concerned because this is causing me a lot of grief.  I'm 
just amazed that this doesn't seem to be happening to anyone else.  Is 
that true?

-- 
Orion Poplawski
Technical Manager                     303-415-9701 x222
NWRA/CoRA Division                    FAX: [phone]
3380 Mitchell Lane                  orion(a)cora.nwra.com
Boulder, CO 80301              http://www.cora.nwra.com

[Eric Sandeen]

Orion Poplawski wrote:
>  I have a lot of problems with rpmdb corruption occurring during
>  (network) installs since around February 2007, and with yum updates 
>  afterwards.  I've reported the problem here 
>  <https://bugzilla.redhat.com/show_bug.cgi?id=230362>;.
>  
>  The problem seems to have gotten worse with the latest rawhide releases
>  - I think about one of every five attempts to perform an install 
>  succeeds.  I'm still somewhat baffled that I appear to be the only 
>  person to be reporting these problems considering how reproducible they 
>  are for me with all kinds of hardware.
>  
>  I keep a separate /var partition, and recently one of our F7 systems' 
>  /var filesystem got corrupted 
>  <https://bugzilla.redhat.com/show_bug.cgi?id=305301>;.  Perhaps the 
>  problems are related.
>  
>  Anyways, I'm concerned because this is causing me a lot of grief.  I'm 
>  just amazed that this doesn't seem to be happening to anyone else.  Is 
>  that true?
>   
I added a note to the bug; can you provide me with an image of the
corrupted filesystem, please?

Thanks,

-Eric

[Lennert Buytenhek]

On Thu, Sep 27, 2007 at 09:01:38AM -0600, Orion Poplawski wrote:

>  I have a lot of problems with rpmdb corruption occurring during
>  (network) installs since around February 2007, and with yum updates 
>  afterwards.  I've reported the problem here 
>  <https://bugzilla.redhat.com/show_bug.cgi?id=230362>;. 
I've been seeing rpmdb environment corruption on ARM systems (all
running >= 2.6.22) for a while now:

	https://www.redhat.com/archives/fedora-arm/2007-August/msg00007.html

The theory was that there is still some problem with shared writable
mmap()s, causing dirty pages to be dropped without being written out
to disk.  db4's data files are exclusively accessed via read()/write(),
but the database environment is exclusively accessed via mmap().

One shred of evidence that seems to point in this direction is that
I haven't seen this corruption occur on any of my ARM systems since
we started mlock()ing the RPM database environment into memory:

	http://fedora-arm.wantstofly.org/diffs-f/rpm-4.4.2.1-10.fc8/rpm-4.4.2-alw...

One theory is that it might show up more on ARM systems because
they typically have less RAM and thus more memory pressure.

[Orion Poplawski]

>  On Thu, Sep 27, 2007 at 09:01:38AM -0600, Orion Poplawski wrote:
>
> > I have a lot of problems with rpmdb corruption occurring during
> > (network) installs since around February 2007, and with yum updates
> > afterwards.  I've reported the problem here
> > <https://bugzilla.redhat.com/show_bug.cgi?id=230362>;.
>
>  One shred of evidence that seems to point in this direction is that
>  I haven't seen this corruption occur on any of my ARM systems since
>  we started mlock()ing the RPM database environment into memory:
>
>
> 	http://fedora-arm.wantstofly.org/diffs-f/rpm-4.4.2.1-10.fc8/rpm-4.4.2-alw...
>  
I re-spun F7 using pungi and rpm-4.4.2.1-1.fc7 rebuilt with this patch. 
Still got corruption on the first yum install after rebooting the
installed system.  This seems pretty common these days.  I'll try
respinning F8 with the equivalent, but I'm not terribly optimistic.

Would I need to rebuild anaconda with the new rpm-devel?  Not sure if I'm
dealing with dynamic or static linking here (I see that rpm-devel does
ship  .a's).

- Orion