Fedora 19 - filesystems slow

[George R Goffe]

Hi,

I have a Fedora 19 system with several large 2-4TB drives attached via USB-SATA docking stations.

I'm seeing a problem whereby a specific filesystem that hasn't been accessed for some time and is REALLY slow to respond to initial requests. ls -alt for example takes a substantial bit of time to respond at first but then response is acceptable after that.

I'm trying to figure out what's happening and why and what I can do about it. It's like the buffers are empty and they need to be filled. Are there kernel tunable parameters that can be changed to avoid this problem?

Thanks,

George...

[Ed Greshko]

On 07/02/14 15:49, George R Goffe wrote:
> have a Fedora 19 system with several large 2-4TB drives attached via USB-SATA docking stations.
>
> I'm seeing a problem whereby a specific filesystem that hasn't been accessed for some time and is REALLY slow to respond to initial requests. ls -alt for example takes a substantial bit of time to respond at first but then response is acceptable after that.
>
> I'm trying to figure out what's happening and why and what I can do about it. It's like the buffers are empty and they need to be filled. Are there kernel tunable parameters that can be changed to avoid this problem?

First, what do you mean by "specific filesystem"?  Second, could your drive just be spun down after a period of inactivity and it takes time to spin up?  USB enclosures often show that characteristic.

[Heinz Diehl]

On 02.07.2014, George R Goffe wrote:
> I'm seeing a problem whereby a specific filesystem that hasn't been
> accessed for some time and is REALLY slow to respond to initial 
> requests.

Looks like there is some power management which puts the device to
sleep/spin down. Try disabling that. Also have a look at
"hdparm -B".

[Rick Stevens]

On 07/02/2014 12:49 AM, George R Goffe issued this missive:
> Hi,
>
> I have a Fedora 19 system with several large 2-4TB drives attached via USB-SATA docking stations.
>
> I'm seeing a problem whereby a specific filesystem that hasn't been accessed for some time and is REALLY slow to respond to initial requests. ls -alt for example takes a substantial bit of time to respond at first but then response is acceptable after that.
>
> I'm trying to figure out what's happening and why and what I can do about it. It's like the buffers are empty and they need to be filled. Are there kernel tunable parameters that can be changed to avoid this problem?

I can think of two things:

1) The drive was spun down because of inactivity. Try looking at the
man page for "hdparm", specifically the "-B" and "-S" options.

2) Caching. If the filesystem hadn't been accessed in a while, its
contents are probably no longer cached. Don't know if you can do
anything about that.
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital    [e-mail] -
- AIM/Skype: therps2        ICQ: 22643734            Yahoo: origrps2 -
-                                                                    -
-              Careful!  Ugly strikes 9 out of 10 people!            -
----------------------------------------------------------------------

[George R Goffe]

Rick,

Thanks for your info. I'll check this out as well. I have used hdparm -B to get and set the APM values... All the drives including the system drive were set to 128. I have set them to 255 now so we'll see what happens.

Actually, the behavior makes me think of a cache that has been emptied... My new activity has to rebuild the cache. Ideally, I'd like to make the cache bigger and have entries written back to the drive(s) BUT remain in cache... A least used algorithm could be used when looking for free space in the cache similar to the one for real memory. Do you know the developers or of a mailing list?

Again, thanks for your response.

George...

________________________________
 From: Rick Stevens 
To: George R Goffe ; Community support for Fedora users  
Sent: Wednesday, July 2, 2014 10:36 AM
Subject: Re: Fedora 19 - filesystems slow

On 07/02/2014 12:49 AM, George R Goffe issued this missive:
> Hi,
>
> I have a Fedora 19 system with several large 2-4TB drives attached via USB-SATA docking stations.
>
> I'm seeing a problem whereby a specific filesystem that hasn't been accessed for some time and is REALLY slow to respond to initial requests. ls -alt for example takes a substantial bit of time to respond at first but then response is acceptable after that.
>
> I'm trying to figure out what's happening and why and what I can do about it. It's like the buffers are empty and they need to be filled. Are there kernel tunable parameters that can be changed to avoid this problem?

I can think of two things:

1) The drive was spun down because of inactivity. Try looking at the
man page for "hdparm", specifically the "-B" and "-S" options.

2) Caching. If the filesystem hadn't been accessed in a while, its
contents are probably no longer cached. Don't know if you can do
anything about that.
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital    [e-mail] -
- AIM/Skype: therps2        ICQ: 22643734            Yahoo: origrps2 -
-                                                                    -
-              Careful!  Ugly strikes 9 out of 10 people!            -

----------------------------------------------------------------------

[Rick Stevens]

On 07/02/2014 11:36 AM, George R Goffe issued this missive:
> Rick,
>
> Thanks for your info. I'll check this out as well. I have used hdparm -B
> to get and set the APM values... All the drives including the system
> drive were set to 128. I have set them to 255 now so we'll see what happens.
>
> Actually, the behavior makes me think of a cache that has been
> emptied... My new activity has to rebuild the cache. Ideally, I'd like
> to make the cache bigger and have entries written back to the drive(s)
> BUT remain in cache... A least used algorithm could be used when looking
> for free space in the cache similar to the one for real memory. Do you
> know the developers or of a mailing list?

The filesystem cache is controlled by the kernel and the filesystem
modules. Part of RAM is reserved for caches and it's shared across all
the mounted filesystems IIRC. If a filesystem goes "inactive", then the
stuff cached for it will eventually be expired and that RAM will be
used for more active filesystems.

I don't know of a way to control either the size of the cache or its
retention period. It might be controllable via a sysctl, but I've never
tried to bugger things like that.

There are a number of resources about individual filesystems out there
on the web. Google your filesystem type and you should find some. For
example, http://ext4.wiki.kernel.org/index.php/Main_Page (for the ext4
filesystem).
> Again, thanks for your response.

No worries, mate. Glad I could help somewhat.
> *From:* Rick Stevens 
> *To:* George R Goffe ; Community support for Fedora
> users 
> *Sent:* Wednesday, July 2, 2014 10:36 AM
> *Subject:* Re: Fedora 19 - filesystems slow
>
> On 07/02/2014 12:49 AM, George R Goffe issued this missive:
> > Hi,
> >
> > I have a Fedora 19 system with several large 2-4TB drives attached
>
> via USB-SATA docking stations.
> > I'm seeing a problem whereby a specific filesystem that hasn't been
>
> accessed for some time and is REALLY slow to respond to initial
> requests. ls -alt for example takes a substantial bit of time to respond
> at first but then response is acceptable after that.
> > I'm trying to figure out what's happening and why and what I can do
>
> about it. It's like the buffers are empty and they need to be filled.
> Are there kernel tunable parameters that can be changed to avoid this
> problem?
>
> I can think of two things:
> The drive was spun down because of inactivity. Try looking at the
>
> man page for "hdparm", specifically the "-B" and "-S" options.
> Caching. If the filesystem hadn't been accessed in a while, its
>
> contents are probably no longer cached. Don't know if you can do
> anything about that.
> Rick Stevens, Systems Engineer, AllDigital [e-mail]
>
>  -
> AIM/Skype: therps2        ICQ: 22643734            Yahoo: origrps2 -
>       -
>          Careful!  Ugly strikes 9 out of 10 people!            -

[Heinz Diehl]

On 02.07.2014, Rick Stevens wrote:
> I don't know of a way to control either the size of the cache or its
> retention period. It might be controllable via a sysctl, but I've never
> tried to bugger things like that.

Look at parameters which can control the kernels virtual memory
management, e.g.:

vm.dirty_background_ratio
vm.dirty_background_bytes
vm.dirty_ratio
vm.dirty_bytes
vm.dirty_writeback_centisecs
vm.dirty_expire_centisecs

If you can reproduce the "slow" disk behaviour by dropping the caches,
maybe you'll gain some advantage by tweaking some of those.

echo 3 > /proc/sys/vm/drop_caches

I for myself have been using

vm.swappiness = 10
 vm.dirty_ratio = 10
 vm.dirty_background_ratio = 5

for a long time, avoiding long stalls when writing dirty pages to
disk.

[Chris Murphy]

On Jul 2, 2014, at 1:49 AM, George R Goffe  wrote:
> Hi,
>
> I have a Fedora 19 system with several large 2-4TB drives attached via USB-SATA docking stations.
>
> I'm seeing a problem whereby a specific filesystem that hasn't been accessed for some time and is REALLY slow to respond to initial requests. ls -alt for example takes a substantial bit of time to respond at first but then response is acceptable after that.
>
> I'm trying to figure out what's happening and why and what I can do about it. It's like the buffers are empty and they need to be filled. Are there kernel tunable parameters that can be changed to avoid this problem?

yum install iotop
iotop -o -d 3

Now access the drive in a manner that reproduces the delay behavior and see what processes are causing io on the drive. What are the mount options for the file system?

Chris Murphy

[poma]

On 02.07.2014 09:49, George R Goffe wrote:
> Hi,
>
> I have a Fedora 19 system with several large 2-4TB drives attached via USB-SATA docking stations.
>
> I'm seeing a problem whereby a specific filesystem that hasn't been accessed for some time and is REALLY slow to respond to initial requests. ls -alt for example takes a substantial bit of time to respond at first but then response is acceptable after that.
>
> I'm trying to figure out what's happening and why and what I can do about it. It's like the buffers are empty and they need to be filled. Are there kernel tunable parameters that can be changed to avoid this problem?
>
> Thanks,
>
> George...

Links to spec:
- mobo
- hdd
- docking station/enclosure

poma