scp over wireless is slow

[Chris Murphy]

Fedora 28 Server
Fedora 28 Workstation
dd-wrt 802.11n Broadcom based router
Connected wirelessly 5GHz, wired ethernet cable is physically
disconnected from Server

File transfer from Workstation to Server

scp: scp itself reports ~620KB/s; where nload on the server reports ~4.9Mbit/s
smb: GNOME reports 4.9MB/s; where nload on the server reports ~39.8Mbit/s

Why? That's rather unexpected.

Command is
scp test.bin f28s.local:/srv/scratch

Using nc, I get speeds slightly faster than smb. OK so encryption? If
I connect wired, and then 'nmcli  c down <ID>' to disconnect the
wireless connection:

scp: 12MB/s, nload ~101Mbit/s
smb: nload ~96Mbit/s

Sooooo, it's not encryption. Why would scp be this much slower only
with a wireless connection? And using:

rync -avzhe ssh test.bin f28s.local:/srv/scratch

Over wireless, this is just as bad as scp.



-- 
Chris Murphy

[Cameron Simpson]

On 02Sep2018 22:16, Chris Murphy <lists(a)colorremedies.com&gt; wrote:
> Fedora 28 Server
> Fedora 28 Workstation
> dd-wrt 802.11n Broadcom based router
> Connected wirelessly 5GHz, wired ethernet cable is physically
> disconnected from Server
>
> File transfer from Workstation to Server
>
> scp: scp itself reports ~620KB/s; where nload on the server reports ~4.9Mbit/s
> smb: GNOME reports 4.9MB/s; where nload on the server reports ~39.8Mbit/s
>
> Why? That's rather unexpected.
>
> Command is
> scp test.bin f28s.local:/srv/scratch
>
> Using nc, I get speeds slightly faster than smb. OK so encryption? If
> I connect wired, and then 'nmcli  c down <ID>' to disconnect the
> wireless connection:
>
> scp: 12MB/s, nload ~101Mbit/s
> smb: nload ~96Mbit/s
>
> Sooooo, it's not encryption. Why would scp be this much slower only
> with a wireless connection? And using:
>
> rync -avzhe ssh test.bin f28s.local:/srv/scratch
>
> Over wireless, this is just as bad as scp. 
There's also compression. And rsync's not the fastest copier around; it's
great 
for incremental change, but sheer data copying is less good.

1: repeat tests with -o compression=no on scp. Though your wired test looks 
like that's not the bottleneck.

2: You're basicly the only person using the wifi during this test? Wifi's 
timesliced; competition might affect this.

3: I presume test.bin is a decent size?

4: Is the speed any different copying in the other direction?

For comparison, I've got a Mac here with an 802.11n 5GHz airport express wired 
to a gigabit ethernet, with an Ubuntu server 2 switches away in the office. Scp 
of an mp4 file from the Mac to the server is telling me about 11.1MB/s right 
now.

My ssh options (which scp will be using) have these settings:

  % ssh -G my-server | egrep -i '(cipher|compre)'
  compression no
  ciphers
[e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],aes256-gcm(a)openssh.com

and an "ssh -v" says this about the encryption:

  debug1: kex: algorithm: curve25519-sha256(a)libssh.org
  debug1: kex: host key algorithm: ecdsa-sha2-nistp256
  debug1: kex: server->client cipher: chacha20-poly1305(a)openssh.com MAC:
<implicit> compression: none
  debug1: kex: client->server cipher: chacha20-poly1305(a)openssh.com MAC:
<implicit> compression: none

Cheers,
Cameron Simpson <cs(a)cskk.id.au&gt;

[Jakub Jelen]

On Sun, 2018-09-02 at 22:16 -0600, Chris Murphy wrote:
>  Fedora 28 Server
>  Fedora 28 Workstation
>  dd-wrt 802.11n Broadcom based router
>  Connected wirelessly 5GHz, wired ethernet cable is physically
>  disconnected from Server
>  
>  File transfer from Workstation to Server
>  
>  scp: scp itself reports ~620KB/s; where nload on the server reports
>  ~4.9Mbit/s
>  smb: GNOME reports 4.9MB/s; where nload on the server reports
>  ~39.8Mbit/s
>  
>  Why? That's rather unexpected.
>  
>  Command is
>  scp test.bin f28s.local:/srv/scratch
>  
>  Using nc, I get speeds slightly faster than smb. OK so encryption? If
>  I connect wired, and then 'nmcli  c down <ID>' to disconnect the
>  wireless connection:
>  
>  scp: 12MB/s, nload ~101Mbit/s
>  smb: nload ~96Mbit/s
>  
>  Sooooo, it's not encryption. Why would scp be this much slower only
>  with a wireless connection? And using:
>  
>  rync -avzhe ssh test.bin f28s.local:/srv/scratch
>  
>  Over wireless, this is just as bad as scp. 

SCP protocol is really slow, especially on networks with high latency
(wireless). The reason why is mostly the size of buffers, which is very
small and SCP waits for every part to be confirmed by the remote host
before sending another part.

You can google "scp speed" and you will get a lot of answers, sometimes
wrongly accusing the encryption or the compression, but really, the RTT
and buffers are the fault as I write here:

https://superuser.com/a/1101203/466930

SCP should be really used only as fast hack for copying files in fast
local networks. For all other cases, use SFTP or rsync if you need
something more complex.

Regards,
-- 
Jakub Jelen
Software Engineer
Security Technologies
Red Hat, Inc.

[Chris Murphy]

On Mon, Sep 3, 2018 at 5:07 AM, Jakub Jelen <jjelen(a)redhat.com&gt; wrote:
>  On Sun, 2018-09-02 at 22:16 -0600, Chris Murphy wrote:
> > Fedora 28 Server
> > Fedora 28 Workstation
> > dd-wrt 802.11n Broadcom based router
> > Connected wirelessly 5GHz, wired ethernet cable is physically
> > disconnected from Server
> >
> > File transfer from Workstation to Server
> >
> > scp: scp itself reports ~620KB/s; where nload on the server reports
> > ~4.9Mbit/s
> > smb: GNOME reports 4.9MB/s; where nload on the server reports
> > ~39.8Mbit/s
> >
> > Why? That's rather unexpected.
> >
> > Command is
> > scp test.bin f28s.local:/srv/scratch
> >
> > Using nc, I get speeds slightly faster than smb. OK so encryption? If
> > I connect wired, and then 'nmcli  c down <ID>' to disconnect the
> > wireless connection:
> >
> > scp: 12MB/s, nload ~101Mbit/s
> > smb: nload ~96Mbit/s
> >
> > Sooooo, it's not encryption. Why would scp be this much slower only
> > with a wireless connection? And using:
> >
> > rync -avzhe ssh test.bin f28s.local:/srv/scratch
> >
> > Over wireless, this is just as bad as scp.
>
>
>  SCP protocol is really slow, especially on networks with high latency
>  (wireless). The reason why is mostly the size of buffers, which is very
>  small and SCP waits for every part to be confirmed by the remote host
>  before sending another part.
>
>  You can google "scp speed" and you will get a lot of answers, sometimes
>  wrongly accusing the encryption or the compression, but really, the RTT
>  and buffers are the fault as I write here:
>
>  https://superuser.com/a/1101203/466930
>
>  SCP should be really used only as fast hack for copying files in fast
>  local networks. For all other cases, use SFTP or rsync if you need
>  something more complex. 
Going from server to workstation, I get double the rate according to
nload. 9.1Mbit/s server to workstation, and 4.9Mbit workstation to
server.

Same results with:
rsync test.bin [e-mail]:~/Downloads

Does that automatically do it over ssh and end up with the same problems?

Separate issue, but related, dd-wrt reports

MAC Address    Interface    Uptime    TX Rate    RX Rate    Info
Signal    Noise    SNR    Signal Quality
00:C2:C6:F0:52:53    wl1    3:07:21    243M    270M    N/A    -49
-89    40    100%
34:02:86:CC:D8:69    wl1    14:33:01135M    135M    N/A    -36    -89
  53    100%

:53 is workstation
:69 is server


The Fedora Workstation Tx and Rx rate is usually 270M but during idle
it's often 6M. This is a:
6c:00.0 Network controller [0280]: Intel Corporation Wireless 8260
[8086:24f3] (rev 3a)

The Fedora Server Tx and Rx rate is never greater than 135M even
though it's closer to the AP. This is a:
02:00.0 Network controller [0280]: Intel Corporation Wireless 3165
[8086:3165] (rev 81)


So for some reason, the server (an Intel NUC) never uses a rate higher
than 135M.

Anyway, in the case where the server is wired, the only 5Ghz wireless
device is the workstation, so no relay. But if the server is wireless,
then both server and workstation are competing and relay is happening.

It might be this old WRT600N is starting to crap out, even though it's
recently reflashed with a current version of dd-wrte. On 2.5Ghz, my
Android phone freaks out every 2-3 days, kernel wake locks prevent the
CPU from idling, and it soaks the battery at a 10%+ rate per hour
doing nothing. Reboot the router? Problem solved for 2 more days. So
does disassociating from that AP. Or airplane mode. Reassociating
without rebooting the router instantly triggers the kernel wake locks.
One of the weirder phone batter problems I've seen.




-- 
Chris Murphy

[Chris Murphy]

On Mon, Sep 3, 2018 at 5:07 AM, Jakub Jelen <jjelen(a)redhat.com&gt; wrote:

>  SCP should be really used only as fast hack for copying files in fast
>  local networks. For all other cases, use SFTP or rsync if you need
>  something more complex. 

I also get the same results with sftp.

sftp chris(a)f28s.local
<authenticate>
sftp> put test.bin

~9MBit/s server to workstation, per nload
~4.9MBit/s workstation to server, per nload

Whether I push or pull the same file with samba, it's 39-44Mbit/s.

I'm using a Fedora install ISO as the test file so it's not really
compressible since most of the payload is already xz compressed.

-- 
Chris Murphy

[Tom Horsley]

On Mon, 3 Sep 2018 13:14:51 -0600
Chris Murphy wrote:

>  I'm using a Fedora install ISO as the test file 
I don't have specific numbers, but I just copied a couple of
linux distro iso files (about 700-800 MB each) to a windows
box I have that uses AC class wireless. Seemed almost as
fast as wired to me. (This was using a windows share hosted
on fedora 28 samba and pulling the files from windows).
Maybe I'm just patient, but I didn't get the impression
of slowness.

[Cameron Simpson]

On 03Sep2018 13:07, Jakub Jelen <jjelen(a)redhat.com&gt; wrote:
> SCP protocol is really slow, especially on networks with high latency
> (wireless). The reason why is mostly the size of buffers, which is very
> small and SCP waits for every part to be confirmed by the remote host
> before sending another part. 
This is categorically false. I've just read through the code to confirm.

The only round trip stuff in scp is at the start and end of the file transfer, 
when it sends the starting file permissions and checks receipt, and at the very 
end. During the transfer it just chucks data at the TCP connection as fast as 
it will accept it. Its internal buffer isn't particularly large, but that is 
irrelevant because (a) the OS reads from files in a sensible fashion and (b) 
TCP colesces writes into the same packet if they arrive fast enough and there's 
room.

So if you're transferring a lot of quite small files, the start/end file 
transaction can get in the way.  But large files go through pretty much at the 
network speed (or the disc speed if the discs are slow and the network is 
fast).

In years of using scp, it has always been pretty fast. And rsync notably slower 
for complete-file copies.

I think Jakub has been misreading the code, probably the atomicio() function, 
which _doesn't_ do an end-to-end delivery of the current buffer; it is just a 
wrapper around the OS read/write call, which may return a short result if its 
underlying buffer empties/fills. That is a _local_ buffer, such as the TCP send 
buffer.

Also, calling a home LAN wireless connection high latency is a bit special 
purpose. It may be higher latency than your wired ethernet, but it is still 
pretty low.

By comparison, I just copied a decent sized file, using scp, over a satellite 
link. Round trip packet time of 600ms-700ms best case. Throughput was 
consistent with my ISP speeds: 5Mbps up, 25Mbps down.

The network packet latency is _not_ a particular issue with scp, because it 
doesn't do the per-buffer end-to-end checks Jakub imagines.

> You can google "scp speed" and you will get a lot of
> answers, sometimes
> wrongly accusing the encryption or the compression, but really, the RTT
> and buffers are the fault as I write here:
>
> https://superuser.com/a/1101203/466930 
I read that. It's a about a paragraph of text with no discussion.

> SCP should be really used only as fast hack for copying files in fast
> local networks. For all other cases, use SFTP or rsync if you need
> something more complex. 
Really, no. Use whatever works best. Scp is fine for large files. For 
incremental change, use rsync (which does a lot of checksum passing to skip 
identical data areas) and for lots of small files use tar or cpio piped over 
ssh, which removes another layer of round trip latency (the per-file sync).

Cheers,
Cameron Simpson <cs(a)cskk.id.au&gt;

[bruce]

On Mon, Sep 3, 2018 at 6:18 PM, Cameron Simpson <cs(a)cskk.id.au&gt; wrote:
>  On 03Sep2018 13:07, Jakub Jelen <jjelen(a)redhat.com&gt; wrote:
> >
> > SCP protocol is really slow, especially on networks with high latency
> > (wireless). The reason why is mostly the size of buffers, which is very
> > small and SCP waits for every part to be confirmed by the remote host
> > before sending another part.
>
>
>  This is categorically false. I've just read through the code to confirm.
>
>  The only round trip stuff in scp is at the start and end of the file
>  transfer, when it sends the starting file permissions and checks receipt,
>  and at the very end. During the transfer it just chucks data at the TCP
>  connection as fast as it will accept it. Its internal buffer isn't
>  particularly large, but that is irrelevant because (a) the OS reads from
>  files in a sensible fashion and (b) TCP colesces writes into the same packet
>  if they arrive fast enough and there's room.
>
>  So if you're transferring a lot of quite small files, the start/end file
>  transaction can get in the way.  But large files go through pretty much at
>  the network speed (or the disc speed if the discs are slow and the network
>  is fast).
>
>  In years of using scp, it has always been pretty fast. And rsync notably
>  slower for complete-file copies.
>
>  I think Jakub has been misreading the code, probably the atomicio()
>  function, which _doesn't_ do an end-to-end delivery of the current buffer;
>  it is just a wrapper around the OS read/write call, which may return a short
>  result if its underlying buffer empties/fills. That is a _local_ buffer,
>  such as the TCP send buffer.
>
>  Also, calling a home LAN wireless connection high latency is a bit special
>  purpose. It may be higher latency than your wired ethernet, but it is still
>  pretty low.
>
>  By comparison, I just copied a decent sized file, using scp, over a
>  satellite link. Round trip packet time of 600ms-700ms best case. Throughput
>  was consistent with my ISP speeds: 5Mbps up, 25Mbps down.
>
>  The network packet latency is _not_ a particular issue with scp, because it
>  doesn't do the per-buffer end-to-end checks Jakub imagines.
>
> > You can google "scp speed" and you will get a lot of answers, sometimes
> > wrongly accusing the encryption or the compression, but really, the RTT
> > and buffers are the fault as I write here:
> >
> > https://superuser.com/a/1101203/466930
>
>
>  I read that. It's a about a paragraph of text with no discussion.
>
> > SCP should be really used only as fast hack for copying files in fast
> > local networks. For all other cases, use SFTP or rsync if you need
> > something more complex.
>
>
>  Really, no. Use whatever works best. Scp is fine for large files. For
>  incremental change, use rsync (which does a lot of checksum passing to skip
>  identical data areas) and for lots of small files use tar or cpio piped over
>  ssh, which removes another layer of round trip latency (the per-file sync).
>
>  Cheers,
>  Cameron Simpson <cs(a)cskk.id.au&gt;
>  _______________________________________________
>  users mailing list -- users(a)lists.fedoraproject.org
>  To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail] 

Hey Cameron,

Sounds like you and Jakub have differing analysis of the issue.

Would you consider posting what you've written to Superuser as well..
to give others a shot at seeing what you've stated. Someone looking to
google on the issue might not see what you posted in this thread.

thanks!

[Cameron Simpson]

On 04Sep2018 09:45, bruce <badouglas(a)gmail.com&gt; wrote:
> Sounds like you and Jakub have differing analysis of the issue.
>
> Would you consider posting what you've written to Superuser as well..
> to give others a shot at seeing what you've stated. Someone looking to
> google on the issue might not see what you posted in this thread. 
Alas, my reputation there is too low to comment. That's what I get for being a 
hermit and hating "forum" websites on the whole.

Cheers,
Cameron Simpson <cs(a)cskk.id.au&gt;