FC 8 & 9: instable file system

[Ralf W. Grosse-Kunstleve]

Randomly, files appear to be missing, leading to application failures. When immediately trying again
the exact same command it (usually) works.

We see this behavior on three systems:

  Fedora 8: 2.6.25.4-10.fc8 #1 SMP Thu May 22 22:58:37 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

  Fedora 8: 2.6.25.6-27.fc8 #1 SMP Fri Jun 13 16:17:54 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

  Fedora 9: 2.6.25.9-76.fc9.x86_64 #1 SMP Fri Jun 27 15:58:30 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

We have other Fedora 8 systems with older kernels that are completely stable, e.g.:

  2.6.23.15-137.fc8 #1 SMP Sun Feb 10 17:03:13 EST 2008 x86_64 x86_64 x86_64 GNU/Linux

We had intermittently missing files on both local disks (ext3) and over NFS, with mounts form various
hosts including a Netapp filer that's serving 30+ other machines without a problem.

My best way of demonstrating the problem is to repeatedly run SCons (which is a make replacement) on
a large source tree with several thousand files. This performs a full dependency analysis and should
just find that everything is already up-to-date. About 10% of the SCons runs fail with "missing" file
warnings, or spurious recompilation, presumably because header files are intermittently missing, which
changes the source signatures.

I already asked the SCons people, the hardware vendor, 3ware support, and I ran yum update a few times
in hopes that the problem goes way. So far it has been persistent.

Has anybody on this list seen similar problems?

Ralf