Anyone know why the NFS write performance with Fedora14 may be slow (without async) ? I have Gigabit networking which is all working fine and the systems in question have been running Fedora in various forms for many years.
Writing a single large file across NFS is fine, about: 32MBytes/sec without async and 51 MBytes/sec with async. However if I untar a 30MByte tar file that has 2466 files in it into an NFS mounted directory with async it takes about 8 seconds but without async it takes over 9 minutes !!
Is this expected ? If not any ideas what is wrong ?
I am seeing the following:
data.bin binary file 100MB jpgraph-3.5.0b1.tar tar archive 29MB 2466 files
# Test1, defaults: nfs version 4, Server /etc/exports: "/data *.kingnet(rw)" Client /etc/fstab: "king.kingnet:/data /data nfs defaults 0 0"
dd if=/tmp/data.bin of=/data/tmp/data.bin bs=102400 32.9 MB/s
dd if=/data/tmp/data.bin of=/tmp/data1.bin bs=102400 66.5 MB/s
time tar -xf /tmp/jpgraph-3.5.0b1.tar real 9m35.235s
# Test2, nfs version 4, async Server /etc/exports: "/data *.kingnet(rw,async)" Client /etc/fstab: "king.kingnet:/data /data nfs defaults 0 0"
dd if=/tmp/data.bin of=/data/tmp/data.bin bs=102400 51.3 MB/s
dd if=/data/tmp/data.bin of=/tmp/data1.bin bs=102400 126 MB/s
time tar -xf /tmp/jpgraph-3.5.0b1.tar real 0m7.938s
# Test3, nfs version 3, async Server /etc/exports: "/data *.kingnet(rw,async)" Client /etc/fstab: "king.kingnet:/data /data nfs defaults,nfsvers=3 0 0"
dd if=/tmp/data.bin of=/data/tmp/data.bin bs=102400 52.6 MB/s
dd if=/data/tmp/data.bin of=/tmp/data1.bin bs=102400 146 MB/s
time tar -xf /tmp/jpgraph-3.5.0b1.tar real 0m4.920s
Terry
Terry Barnaby <terry1 <at> beam.ltd.uk> writes:
...
It would be useful to publish a test for # Test4, defaults: nfs version 3 that is with "sync" option, so we could see if similar degradation was present with older protocol ?
One other thing: it would be interesting to see the results if instead of in-place (/data/...) execution of time tar -xf ...
you first untared your files into non-NFS dir mkdir -p /tmp/untared cd /tmp/untared tar -xf /tmp/jpgraph-3.5.0b1.tar
and then executed 'time cp /tmp/untared/* /data/tmp/'
JB
On 09/25/2011 04:39 PM, JB wrote:
Terry Barnaby<terry1<at> beam.ltd.uk> writes:
...
It would be useful to publish a test for # Test4, defaults: nfs version 3 that is with "sync" option, so we could see if similar degradation was present with older protocol ?
One other thing: it would be interesting to see the results if instead of in-place (/data/...) execution of time tar -xf ...
you first untared your files into non-NFS dir mkdir -p /tmp/untared cd /tmp/untared tar -xf /tmp/jpgraph-3.5.0b1.tar
and then executed 'time cp /tmp/untared/* /data/tmp/'
JB
Thanks for the reply. The extra test data is:
# Test1, defaults: nfs version 4, sync Server /etc/exports: "/data *.kingnet(rw)" Client /etc/fstab: "king.kingnet:/data /data nfs defaults 0 0"
dd if=/tmp/data.bin of=/data/tmp/data.bin bs=102400 32.9 MB/s
dd if=/data/tmp/data.bin of=/tmp/data1.bin bs=102400 66.5 MB/s
time tar -xf /tmp/jpgraph-3.5.0b1.tar real 9m35.235s
time cp -a /tmp/jpgraph-3.5.0b1 /data/tmp real 6m6.373s
# Test4, nfs version 3, sync Server /etc/exports: "/data *.kingnet(rw,async)" Client /etc/fstab: "king.kingnet:/data /data nfs defaults,nfsvers=3 0 0"
dd if=/tmp/data.bin of=/data/tmp/data.bin bs=102400 34.6 MB/s
dd if=/data/tmp/data.bin of=/tmp/data1.bin bs=102400 120 MB/s
time tar -xf /tmp/jpgraph-3.5.0b1.tar real 4m20.355s
time cp -a /tmp/jpgraph-3.5.0b1 /data/tmp real 6m8.394s
Basically NFS with sync for NFS version 3 and 4 is very slow, NFSV4 seems slowest.
Note that CPU usage on both client and server is virtually nill during the tests (server waiting is for I/O 48% on dual core system). The server has 2 x SATA disks in RAID-1 for /data. I wonder if NFS is doing a complete sync() to disk on each file close ??
Cheers
Terry
Are you using userspace NFS or the kernel NFS? The kernel NFS _should_ be faster.
On Sun, Sep 25, 2011 at 10:02 AM, Terry Barnaby terry1@beam.ltd.uk wrote:
I wonder if NFS is doing a complete sync() to disk on each file close ??
If you are using the userspace NFS, the strace command will show all the system calls it makes. Run strace against the NFS daemon and untar a small tarball with just a few files, otherwise it will spew mountains of logs.
strace -f will trace child processes in the event that the daemon forks when you use it.
On 09/25/2011 06:13 PM, Don Quixote de la Mancha wrote:
Are you using userspace NFS or the kernel NFS? The kernel NFS _should_ be faster.
On Sun, Sep 25, 2011 at 10:02 AM, Terry Barnabyterry1@beam.ltd.uk wrote:
I wonder if NFS is doing a complete sync() to disk on each file close ??
If you are using the userspace NFS, the strace command will show all the system calls it makes. Run strace against the NFS daemon and untar a small tarball with just a few files, otherwise it will spew mountains of logs.
strace -f will trace child processes in the event that the daemon forks when you use it.
As far as I was aware kernel space NFS has been the default for many years. Will check to see if somehow user space NFS is being used ...
Terry Barnaby <terry1 <at> beam.ltd.uk> writes:
... # Test1, defaults: nfs version 4, sync Server /etc/exports: "/data *.kingnet(rw)" Client /etc/fstab: "king.kingnet:/data /data nfs defaults 0 0"
dd if=/tmp/data.bin of=/data/tmp/data.bin bs=102400 32.9 MB/s
dd if=/data/tmp/data.bin of=/tmp/data1.bin bs=102400 66.5 MB/s <+++++++++++++++++++++++++++++++++++++++++++++++++++
time tar -xf /tmp/jpgraph-3.5.0b1.tar real 9m35.235s <===========================================
time cp -a /tmp/jpgraph-3.5.0b1 /data/tmp real 6m6.373s
# Test4, nfs version 3, sync Server /etc/exports: "/data *.kingnet(rw,async)" Client /etc/fstab: "king.kingnet:/data /data nfs defaults,nfsvers=3 0 0"
dd if=/tmp/data.bin of=/data/tmp/data.bin bs=102400 34.6 MB/s
dd if=/data/tmp/data.bin of=/tmp/data1.bin bs=102400 120 MB/s <+++++++++++++++++++++++++++++++++++++++++++++++++++
time tar -xf /tmp/jpgraph-3.5.0b1.tar real 4m20.355s <===========================================
time cp -a /tmp/jpgraph-3.5.0b1 /data/tmp real 6m8.394s ...
The results marked (<===, <+++) show a regression for NFS v4 vice NFS v3, a decline of 100% for each test case. I think you should let them know (BZ report) - the type of test performed would be of interest to them.
I would suggest this as well: to test by eliminating a possible problem with network, you could perform NFS tests (as above) locally (on your client machine in question), that is, install NFS server and client software locally, export some dir locally, and mount the exported dir locally. That is, you would mostly test the NFS software only.
JB
Try writing to a different type of filesystem. It might be the filesystem's fault. MacTCP on the Classic Mac OS got a real bad rap because FTP writes were very slow, but it was easy to show that the problem was in the Heirarchical Filesystem.
Is your destination filesystem journaled? Maybe flushing the journal is causing your hit.
On 09/25/2011 07:03 PM, Don Quixote de la Mancha wrote:
Try writing to a different type of filesystem. It might be the filesystem's fault. MacTCP on the Classic Mac OS got a real bad rap because FTP writes were very slow, but it was easy to show that the problem was in the Heirarchical Filesystem.
Is your destination filesystem journaled? Maybe flushing the journal is causing your hit.
I don't think it is the file systems fault, at least not directly, NFS with async is fine. Both systems are using ext4, so they are journaled.
Cheers
Terry
On 09/25/2011 06:41 PM, JB wrote:
Terry Barnaby<terry1<at> beam.ltd.uk> writes:
... # Test1, defaults: nfs version 4, sync Server /etc/exports: "/data *.kingnet(rw)" Client /etc/fstab: "king.kingnet:/data /data nfs defaults 0 0"
dd if=/tmp/data.bin of=/data/tmp/data.bin bs=102400 32.9 MB/s
dd if=/data/tmp/data.bin of=/tmp/data1.bin bs=102400 66.5 MB/s<+++++++++++++++++++++++++++++++++++++++++++++++++++
time tar -xf /tmp/jpgraph-3.5.0b1.tar real 9m35.235s<===========================================
time cp -a /tmp/jpgraph-3.5.0b1 /data/tmp real 6m6.373s
# Test4, nfs version 3, sync Server /etc/exports: "/data *.kingnet(rw,async)" Client /etc/fstab: "king.kingnet:/data /data nfs defaults,nfsvers=3 0 0"
dd if=/tmp/data.bin of=/data/tmp/data.bin bs=102400 34.6 MB/s
dd if=/data/tmp/data.bin of=/tmp/data1.bin bs=102400 120 MB/s<+++++++++++++++++++++++++++++++++++++++++++++++++++
time tar -xf /tmp/jpgraph-3.5.0b1.tar real 4m20.355s<===========================================
time cp -a /tmp/jpgraph-3.5.0b1 /data/tmp real 6m8.394s ...
The results marked (<===,<+++) show a regression for NFS v4 vice NFS v3, a decline of 100% for each test case. I think you should let them know (BZ report) - the type of test performed would be of interest to them.
I would suggest this as well: to test by eliminating a possible problem with network, you could perform NFS tests (as above) locally (on your client machine in question), that is, install NFS server and client software locally, export some dir locally, and mount the exported dir locally. That is, you would mostly test the NFS software only.
JB
I don't think it is a network issue. iperf shows this as Ok as do the NFS tests with async set. Anyway a test with local nfs: time tar -xf /tmp/jpgraph-3.5.0b1.tar real 3m27.320s
The local disk access speeds in both systems are fine. time (tar -xf /tmp/jpgraph-3.5.0b1.tar; sync) real 0m2.816s
Cheers
Terry
On 09/25/2011 05:45 AM, Terry Barnaby wrote:
Anyone know why the NFS write performance with Fedora14 may be slow (without async) ? I have Gigabit networking which is all working fine and the systems in question have been running Fedora in various forms for many years.
I found NFSv4 to be completely unusable on my F14 systems. I could never figure out where the problem was. It was unusable on my F14 clients when I had a F13 server, and once I upgraded the server to F14, my F13 clients became unusable...
In desperation, I had to move to glusterfs, which fixed all my problems:
http://www.mail-archive.com/users@lists.fedoraproject.org/msg40298.html
I've been very happy with it since the forced migration! It has a lot fewer oddities than NFS, and it handles sqlite/firefox file-locking just fine, unlike some (all?) NFS variants.
- Mike
On 09/26/2011 01:08 PM, Dr. Michael J. Chudobiak wrote:
On 09/25/2011 05:45 AM, Terry Barnaby wrote:
Anyone know why the NFS write performance with Fedora14 may be slow (without async) ? I have Gigabit networking which is all working fine and the systems in question have been running Fedora in various forms for many years.
I found NFSv4 to be completely unusable on my F14 systems. I could never figure out where the problem was. It was unusable on my F14 clients when I had a F13 server, and once I upgraded the server to F14, my F13 clients became unusable...
In desperation, I had to move to glusterfs, which fixed all my problems:
http://www.mail-archive.com/users@lists.fedoraproject.org/msg40298.html
I've been very happy with it since the forced migration! It has a lot fewer oddities than NFS, and it handles sqlite/firefox file-locking just fine, unlike some (all?) NFS variants.
- Mike
Hi,
Thanks for the info. Might try that, but I still need NFS for other systems and having used it for almost 30 years am a bit used to it ! Actually I haven't really had any real problems with NFSv4 under F13 or F14 both at work and home (each network one server and about 6 clients mounting /home and /data). The only real issue has been performance over an OpenVPN connection over ADSL, but that is not too surprising (although an ls -l takes significantly longer that it really should) and this performance issue when writing multiple files.
Does gluserfs support client side file and attribute caching (cachefilesd is used with NFS) ? I use that with NFS over the OpenVPN/ADSL link which helps a bit (although I think it should work much better than it actually does).
Cheers
Terry
On 09/26/2011 04:46 PM, Terry Barnaby wrote:
Thanks for the info. Might try that, but I still need NFS for other systems and having used it for almost 30 years am a bit used to it !
glusterfs and nfs mix nicely - you can use the native glusterfs client (I do), or you can use an nfs client to access the glusterfs server.
You can export a folder using both an nfs server and a glusterfs server at the same time, which is nice for a gradual migration.
mounting /home and /data). The only real issue has been performance over an OpenVPN connection over ADSL, but that is not too surprising (although an ls -l takes significantly longer that it really should) and this performance issue when writing multiple files.
I use glusterfs over openvpn + cable modem, and it seems much more responsive than nfs ever was (judged using "ls -l"). But I haven't benchmarked it objectively.
Does gluserfs support client side file and attribute caching (cachefilesd is used with NFS) ?
I'm not sure about that...
Anyway... it sure solved a lot of stability problems for me. glusterfs deserves to be better known as a reasonable NFS alternative.
- Mike
-
Don Quixote de la Manvha -
Dr. Michael J. Chudobiak -
JB -
Terry Barnaby