Why does disk I/O slow down a CPU bound task?

[M. Edward (Ed) Borasky]

On Mon, Mar 30, 2015 at 12:58 PM, Dave Johansen <davejohansen at gmail.com> wrote:
> I noticed on RHEL 6 that when a large amount of disk I/O is happening that
> CPU bound tasks "slow down". I have been able to reproduce it in Fedora 21
> as well and here are the instructions of how I can reproduce it with a
> simple test:
>
> 1) Build the disk_test.cc (the "CPU bound task") and run it.
> 2) Create a large file to copy ( fallocate -l 10G junk ).
> 3) Copy that file with a one minute delay between copies ( while true; do cp
> junk junk2; sleep 60; done )
>
> If you direct the output of disk_test.cc to a file, then you can plot the
> results in gnuplot with the following commands to see the change in the mean
> time between "finishing the work cycle" when the file is being copied:
> set xdata time
> set timefmt "%s"
> plot "out.txt" using 1:3 with lines
>
> You can also notice that the load average is also going up, so it seems like
> something in the kernel/scheduler is getting some sort of exclusive lock in
> the disk I/O process and that's causing the CPU bound task to not be able to
> execute when it should. Any ideas?
>
> Thanks,
> Dave
>
> --
> devel mailing list
> devel at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/devel
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

This has been an issue in Linux for a long time. It was old news in
2008 when I dug into it. Basically, the response you will get when you
say "It hurts when I do that" is "Don't do that!" ;-)

As a well-known performance engineer once said, "The only good I/O is
a dead I/O". Cache as if your life depended on it, use SSDs when you
can afford them, etc. If you *must* build on top of rotating disks,
there are tools to tune the I/O schedulers - blktrace and systemtap
are the best ones at the low level.

By the way, the Linux "load average" counts both CPU-bound tasks ("R"
state) and I/O-bound tasks ("D" state). That's why the load average is
going up. But the cure is to not design I/O-bound applications. ;-)

-- 
OSJourno: Robust Power Tools for Digital Journalists
http://www.znmeb.mobi/stories/osjourno-robust-power-tools-for-digital-journalists

Remember, if you're traveling to Bactria, Hump Day is Tuesday and Thursday.