Ethernet monopolizing LAN9514 upstream USB bus

[Bernardo Donadio]

Is there a way to stop the Ethernet controller from monopolizing the 
upstream USB channel, while in high usage, in detriment of other USB 
devices on LAN9514 (Raspberry Pi 2 USB-Hub/Ethernet chip)?

Basically, if you have a USB-attached storage device and start writing 
data that comes from the Ethernet into this USB disk, the Ethernet chip 
will always have priority on the USB bus and the kernel will never have 
the chance to use it to write on the storage device.

As a result, the data incoming from the Ethernet gets stored in the 
kernel IO buffer until memory runs out, and then the transfer stops for 
a while (as the kernel starting dropping TCP packets and the TCP flow 
control algorithm kicks in). On the other hand, if you stop the network 
transfer before the buffer is filled, the content on it is immediately 
flushed to disk.

This would not be a huge problem if the USB-attached storage devices 
didn't have a timeout: after not receiving some keepalive packets from 
the system it's attached to, it dies. With my USB disk, at least, you 
have to unplug its USB cable and plug it again to convince its 
controller that the system is back up.

Of course, this is a even bigger issue on Raspberry if the system's root 
is installed on this USB disk.

I'm not sure that this is fixed in the Raspberry Pi Foundation's remix 
of Debian, but I'm definitely hitting this bug on Fedora 25 beta with 
Linux 4.8.0-0.rc7.git0.1.fc25.armv7hl.

How to reproduce, considering /dev/sda is some USB-attached storage on 
the rasp2:
desktop$ dd if=some_big_file bs=8M | ssh root@rasp2 dd of=/dev/sda bs=8M

Meanwhile, you may watch the kernel buffer's filling up on the rasp2 
with htop or free. You may also notice that if you cancel the SSH 
transfer before the rasp2 kernel buffer is filled (and also before your 
USB disk gives up waiting on the keepalive packet), the content stored 
on the buffer is almost immediately flushed to the disk.

How can we fix this?

-- 

Bernardo Donadio
DevOps Engineer at Alligo Tecnologia
https://bcdonadio.com/

[Peter Robinson]

On Sun, Oct 23, 2016 at 7:36 AM, Bernardo Donadio
<bcdonadio(a)bcdonadio.com&gt; wrote:
>  Is there a way to stop the Ethernet controller from monopolizing the
>  upstream USB channel, while in high usage, in detriment of other USB devices
>  on LAN9514 (Raspberry Pi 2 USB-Hub/Ethernet chip)? 
Not really. You might try putting it into polling mode as opposed to
the standard interrupt mode but I'm not even sure the usb NICs would
support that.

>  Basically, if you have a USB-attached storage device and start
> writing data
>  that comes from the Ethernet into this USB disk, the Ethernet chip will
>  always have priority on the USB bus and the kernel will never have the
>  chance to use it to write on the storage device.
>
>  As a result, the data incoming from the Ethernet gets stored in the kernel
>  IO buffer until memory runs out, and then the transfer stops for a while (as
>  the kernel starting dropping TCP packets and the TCP flow control algorithm
>  kicks in). On the other hand, if you stop the network transfer before the
>  buffer is filled, the content on it is immediately flushed to disk.
>
>  This would not be a huge problem if the USB-attached storage devices didn't
>  have a timeout: after not receiving some keepalive packets from the system
>  it's attached to, it dies. With my USB disk, at least, you have to unplug
>  its USB cable and plug it again to convince its controller that the system
>  is back up.
>
>  Of course, this is a even bigger issue on Raspberry if the system's root is
>  installed on this USB disk.
>
>  I'm not sure that this is fixed in the Raspberry Pi Foundation's remix of
>  Debian, but I'm definitely hitting this bug on Fedora 25 beta with Linux
>  4.8.0-0.rc7.git0.1.fc25.armv7hl. 
Well I have no interest in doing kernel archaeology on a fork of an
old kernel. we use the upstream kernel for a reason. You probably want
to "dnf upgrade" to the latest 4.8.3 kernel although I doubt it'll
have any effect.

>  How to reproduce, considering /dev/sda is some USB-attached storage
> on the
>  rasp2:
>  desktop$ dd if=some_big_file bs=8M | ssh root@rasp2 dd of=/dev/sda bs=8M
>
>  Meanwhile, you may watch the kernel buffer's filling up on the rasp2 with
>  htop or free. You may also notice that if you cancel the SSH transfer before
>  the rasp2 kernel buffer is filled (and also before your USB disk gives up
>  waiting on the keepalive packet), the content stored on the buffer is almost
>  immediately flushed to the disk.
>
>  How can we fix this? 
We probably can't, but ultimately the RPi has a single, some what
buggy, usb interface with a hub attached, it's not designed for high
throughput data IO workloads. You're probably better off getting a
device that's more suited to those sorts of workloads. Even the Orange
Pi series of devices are better in this regard as they have a
dedicated SoC attached NIC and 3 usb root devices.

Peter

[Bernardo Donadio]

On 10/23/2016 08:21 AM, Peter Robinson wrote:
> > How can we fix this?
>
>  We probably can't, but ultimately the RPi has a single, some what
>  buggy, usb interface with a hub attached, it's not designed for high
>  throughput data IO workloads. You're probably better off getting a
>  device that's more suited to those sorts of workloads. Even the Orange
>  Pi series of devices are better in this regard as they have a
>  dedicated SoC attached NIC and 3 usb root devices. 
Well, ok. I totally understand that this is a somewhat hacky design, 
with inherent flaws. Anyway, the caveat is now documented for future 
Google lurkers. :)

Thanks.
-- 
Bernardo Donadio
DevOps Engineer at Alligo Tecnologia
https://bcdonadio.com/