I/O Scheduling results in poor responsiveness

[Nathan Grennan]

     Why is the command below all that is needed to bring the system to 
it's knees? Why doesn't the io scheduler, CFQ, which is supposed to be 
all about fairness starve other processes? Example, if I open a new file 
in vim, and hold down "i" while this is running it will pause the 
display of new "i"s for seconds, sometimes until the dd write is 
completely finished. Another example is applications like firefox, 
thunderbird, xchat, and pidgin will stop refreshing for 10+ seconds.

  dd if=/dev/zero of=test-file bs=2M count=2048

  I understand the main difference between using oflag=direct or not 
relates to if the io scheduler is used, and if the file is cached or 
not. I can see this clearly by watching cached rise without 
oflag=direct, stay the same with it, and go way down when I delete the 
file after running dd without oflag=direct.

  The system in question is running Fedora 8. It is an E6600, 4gb 
memory, and 2x300gb Seagate sata drives. The drives are setup with md 
raid 1, and the filesystem is ext3. But I also see this with plenty of 
other systems with more cpu, less cpu, less memory, raid, and no raid.
 
  I have tried various tweaks to sys.vm settings, tried changing the 
scheduler to as or deadline. Nothing seem to get it to behave, other 
than oflag=direct.

  Using dd if=/dev/zero is just an easy test case.  I see this when 
copying large files, creating large files, and using virtualization 
software that does heavy i/o on large files.


 
  The command below seems to result in cpu idle 0 and io wait 100%. As 
shown by "vmstat 1"

dd if=/dev/zero of=test-file bs=2M count=2048

2048+0 records in
2048+0 records out
4294967296 bytes (4.3 GB) copied, 94.7903 s, 45.3 MB/s


  The command below seems to work much better for responsiveness. The 
cpu idle will be around 50, and the io wait will be around 50.

dd if=/dev/zero of=test-file2 bs=2M count=2048 oflag=direct

2048+0 records in
2048+0 records out
4294967296 bytes (4.3 GB) copied, 115.733 s, 37.1 MB/s