On Fri, 7 Sep 2018 13:22:35 -0700 Rick Stevens <ricks(a)alldigital.com> wrote:
On 09/07/2018 12:48 PM, jdow wrote:
> On 20180907 11:49, Ranjan Maitra wrote:
>> On Fri, 7 Sep 2018 14:26:32 -0400 Tim Evans <tkevans(a)tkevans.com> wrote:
>>> On 09/07/2018 02:22 PM, Ranjan Maitra wrote:
>>>> I was doing this and it is definitely faster than rsync:
>>>> cd /drive1
>>>> tar cf - uncopieddir1 uncopieddir2 ... | ( cd /drive2 ; tar xf - )
>>>> But, after about 16 hours, I am only 229G in (out of 3.7T). This is
>>>> much slower than the other thread with USB drives which did 400GB in
>>>> 8 hours.
>>> Try this:
>>> # cd /drive1
>>> # find . -print -depth | cpio -pdm /drive2
>> So, should I cancel the other job and do this? I am not sure what to
>> do, sorry.
> Unless the drive is sparsely filled plain old "dd" may be the quickest
> end to end operation. And you get an exact duplicate so the OS and
> filesystem on the disks won't give you any trouble if you have a mix of
> dd if=/dev/<drive1> of=/dev/<drive2> bs=1073741824 conv=sparse,noerror
> Remember that "kill -USR1 $pid" will then tell you how much you've
> copied if you get "is done yet" anxiety.
> You suffer no time for directory accesses, file reads and writes, and
> all that nonsense. If you do this fairly often keep it handy as a script
> or at least an example. The only time this won't work as nicely is if
> the source and destination disks are not on the same machine. The
> slowness of the tar solution won't be a problem because the network will
> as likely as not be your limiting factor.
Good solution, but I believe he's really hitting the USB write
speed/cache flush/bus contention issue. I doubt that filesystem overhead
has anything to do with his problem. IIRC, he's doing USB---->USB
This is why, when possible, I use an ESATA external drive for backups.
I try to buy laptops that have ESATA ports and I have ESATA PCIe cards
for my desktops. It's not ideal, but I've seen far too many cases where
it's simply the USB ports not keeping up. It'd be interesting for him
to install gkrellm and monitor both of the drives involved to see which
one is stalling. I'm willing to bet it's the target (write) drive (the
source drive will stop doing I/O while the target drive has I/O out the
But I've been wrong before (as any reader of this forum can attest to).
I have SATA drives, both internal. No USB involved. You are mixing up with the other