kde-freezes while large USB copy

[Darren Steven]

I can explain this, as I see it too. Well, I can partially explain.

It's caused by the kernel allocating large write buffers that can represent
many minutes (or hours) of write time for a slow USB device. You copy that
file from a higher performing storage device, and you end up with many GB
buffered writes. KDE (or something in the UI render path) decides to stop
while doing an fsync ( I'm not sure exactly what), and you need to wait for
that to complete. Worst case for me was filling a class 4 micro SD with
32GB of music. My machine has 16GB of RAM, and a quite fast HDD, so it soon
had a few GB queued, which drains at about 1MB/sec (due to some issues on
the other end of the USB, with a similar root cause).

2 work arounds I found:

1. If it's a FAT filesystem (as your USB stick may be), mount it with the
sync option, which can be done with something like this (after KDE has
mounted):
udisks --mount /dev/disk/by-uuid/XXXXX --mount-options sync

It's slow, but at least you can work while it copies

2. Limit the kernel write buffers to some appropriate value, using
/proc/sys/vm/dirty_background_bytes and/or dirty_bytes. This is a bit
messy, and has some side-effects on other drive performance, so you trade
latency for throughput. . There is also dirty_background_ratio and
dirty_ratio - you use bytes or ratio, setting one clears the other, but
they basically set thresholds as to when the kernel will start flushing,
and when user process blocks while write flushes (I think).

Apparently it defaults (as a ratio) to about 20% of RAM, which can be a lot
these days, and it also has some minimum value, which is still very large
for a 16GB machine (or even a 4GB machine). For me, something on the order
of 8MB was OK for that slow flash, but that killed the HDD raid I have. I
currently run dirty_bytes at 128MB and dirty_background_bytes at 32MB. It
still kills me occasionally, but has minimal impact on the HDD performance.
128MB is about 2x the disks internal buffer in my case.

It's probably a default Fedora may want to tweak based on machine config,
and I understand there has been some healthy discussion in kernel lists
about it. To me it seems the kernel needs to become acutely aware of what
the real throughput of a device is, not what the device claims, and allow
the user to set minimum write latency. Almost like tcp's windowing system
(with slow start). Filesystems matter too apparently, and ext4 is renowned
for latency issues (btrfs more so)

Cheers

Darren


2011/11/20 Reindl Harald <h.reindl at thelounge.net>

>
>
> Am 20.11.2011 03:38, schrieb Richard Heck:
> > On 11/19/2011 11:16 AM, Reindl Harald wrote:
> >> Am 19.11.2011 17:09, schrieb Martin Kho:
> >>> On Saturday 19 November 2011 08:00:43 Tayfun Kayhan wrote:
> >>>> +1 ! Mine is sometimes completely frozen and nothing can come it back
> :/ But
> >>>> mostly, openning System Activity with Ctrl + Esc is fixin it, as a
> >>>> workaround.
> >>> On the test list there was a threat with a comparable issue, see [1].
> May be
> >>> it's the same as what you see.
> >>>
> >>> [1]
> http://lists.fedoraproject.org/pipermail/test/2011-November/104368.html
> >> i can not say this in general
> >>
> >> only if the I/O goes to USB2 or USB3 while the last seems
> >> to have a kernel-bug with my controller and tends to damage
> >> filesystmes and lsoe connection but thats off-topic
> >>
> > I've seen something like this, too, but am not sure if it is KDE-related
> > or controller related. But trying to copy lots of data to a USB2 stick
> > here is very, very slow. It works at first just fine, but then slows
> > down as things proceed. Copying, say, 4GB can take 2-3 hours. I guess I
> > should at least try it under another WM.
>
> i am also not sure if other WM would show a different behavior
>
> that it slows down until buffers are full is clear
> but what is not acceptable is that the whole computer in a class
> few years ago clusters did not have is unuseable while data transfer
> to an usb-drive while this never happens to eSATA, but not uch drives
> have eSATA :-(
>
>
> _______________________________________________
> kde mailing list
> kde at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/kde
> New to KDE4? - get help from http://userbase.kde.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.fedoraproject.org/pipermail/kde/attachments/20111120/bb502daf/attachment-0001.html