I apologize for not more closely following this thread; if this has already been suggested, please just take it as reinforcement.
Once, long ago--actually, on Fri, Jun 21, 2013 at 05:09:09PM CDT--jdow (jdow@earthlink.net) said:
Paul, when disks start throwing that error it's typically "months" or less before it does something unfriendly like failing to spin up.
Or less. Remember--when it's out of sectors for remapping, it's near the end.
Put a replacement drive high on your list of things to do.
Not only that--given that disks are (relatively) cheap, and data is expensive, this would be the time to consider putting in at least a RAID1 array. Life gets a lot less hectic when a disk failure is only one in an array.
I just went through this--a mdadm RAID5 with three disks, and one of my disks finally failed. It was just a matter of getting another disk at the best price, while the server limped along on two drives. And limping is only relative--there was no appreciable degradation in performance.
Replacing the disk took about a half-hour, and most of that was physical remove/replace effort. Told it about the new disk, and after a few hours it had rebuilt the array. No muss, no fuss.
Cheers, -- Dave Ihnat President, DMINET Consulting, Inc. dihnat@dminet.com