Copy with SCP - low performance

[Jeff Vian]

Christopher Ness wrote:

>On Mon, 2004-03-29 at 18:04, Horacio Reyes wrote:
>  
>
>>I am using scp to copy large amount of small files between 2 servers
>>with a firewall in the middle, but the copy takes all CPU, this is the
>>situation:
>>
>>- The files are copied in batch of 100 tif images, in a directory with
>>more than 5 million files.
>>    
>>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You need to use a method that can properly handle that many files.
I hope you do not need to copy them ALL very often.


>Do you have the diskspace to make a tarball and optionally compress the
>tarball before sending?
>
>I'm not sure if it's the small files or the firewall that is causing
>your slow transfer.  If it's the number of files this could be a
>solution.
>
>Cheers,
>Chris
>  
>