ext3 filesystem performance issues

[Bruno Wolff III]

On Wed, Sep 12, 2007 at 20:07:42 -0400,
  Matthew Flaschen <matthew.flaschen at gatech.edu> wrote:
> > 
> > I have directories with several million files in them.
> 
> Just curious...what for?

Legacy, they need to be fixed.

I am tracking IP addresses that sent me spam and blocking them and I save
a file for each IP address with the headers in it in case I want to check
on things later. When I started doing that it took a long time to get
hit bit 100K unique addresses. Now it takes a couple of months to get hit
by over a million unique addresses. I don't check as often as I used to because
even doing ls -f | wc takes several minutes to run. Not too mention that each
file takes up 8K of disk space on ext3 file systems.
I have been meaning to rewrite my tracking system to put everything in a
single (or perhaps several) files to make dealing with the data less
burdensome. I just haven't gotten round to it yet.