2:50 p.m.
I started a tar command from
one external eSATA drive (ext4), connected to eSATA port,
out to a USB flash drive (with vfat). The USB stick is touted
to support 50MB/s write, 160MB/s read.
The USB port itself is USB 2.0 (480MBits/s).
The desktop (Mate) response goes down the tubes.
I ran top and found that cpu is only 3% usage by firefox.
rest are a few at 1%, and rest showing 0%.
Load average: load average: 5.95, 5.81, 6.05
Wow!!!! Why?
I ran iotop -d 3 and found
3263 be/4 root 0.00 B/s 0.00 B/s 0.00 % 99.99 % [kworker/u16:3]
61 be/4 root 0.00 B/s 0.00 B/s 0.00 % 99.99 % [kswapd0]
3180 be/4 root 335.95 B/s 3.62 M/s 0.00 % 99.99 % tar -C
/sdc1 -xpvf -
4071 be/4 root 0.00 B/s 0.00 B/s 0.00 % 27.36 % [kworker/0:2]
Processes that are started but idle:
1 instance of smplayer (idle - nothing being played)
2 instances of TB (for 2 different profiles) - idle - no incoming or
outgoing emails.
both check for mail every so many minutes.
1 instance of FF with a total of 6 tabs - all of them idle (no videos,
no audios on them
and no animations.).
So, it seems that the scheduler is letting the disk io pretty much
commandeer and hijack system response time for the DT.
Is there a configuration tweak to grant more cycles to the DT (mouse,
KB) interaction
with other GUI based programs?
4:13 p.m.
On Mon, 2015-07-13 at 13:50 -0600, jd1008 wrote:
I started a tar command from
one external eSATA drive (ext4), connected to eSATA port,
out to a USB flash drive (with vfat). The USB stick is touted
to support 50MB/s write, 160MB/s read.
Have you actually measured its real write speed for large files? Try
something like:
time dd if=/dev/zero of=/the/usb/drive count=1000 bs=1M
You might be surprised.
poc
4:37 p.m.
On 07/13/2015 03:13 PM, Patrick O'Callaghan wrote:
On Mon, 2015-07-13 at 13:50 -0600, jd1008 wrote:
> I started a tar command from
> one external eSATA drive (ext4), connected to eSATA port,
> out to a USB flash drive (with vfat). The USB stick is touted
> to support 50MB/s write, 160MB/s read.
Have you actually measured its real write speed for large files? Try
something like:
time dd if=/dev/zero of=/the/usb/drive count=1000 bs=1M
You might be surprised.
poc
Well, I am not surprised, but disappointed (seriously!!!)
$ time dd if=/dev/zero of=/run/media/jd/USB3_ST1-2K/test-wr-speed.txt
bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 57.6116 s, 18.2 MB/s
real 0m57.88s
user 0m0.00s
sys 0m1.28s
So much for 50MB/s write speed!!!!
Testing the read speed:
$ time dd if=/run/media/jd/USB3_ST1-2K/F3296296960 of=/dev/null bs=8M
392+1 records in
392+1 records out
3296296960 bytes (3.3 GB) copied, 106.6 s, 30.9 MB/s
real 1m46.62s
user 0m0.00s
sys 0m4.48s
So much for 160MB/s read.
I am going to diss this piece of ... [expletive/deleted]
6:09 p.m.
On Mon, 2015-07-13 at 15:37 -0600, jd1008 wrote:
On 07/13/2015 03:13 PM, Patrick O'Callaghan wrote:
> On Mon, 2015-07-13 at 13:50 -0600, jd1008 wrote:
> > I started a tar command from
> > one external eSATA drive (ext4), connected to eSATA port,
> > out to a USB flash drive (with vfat). The USB stick is touted
> > to support 50MB/s write, 160MB/s read.
> Have you actually measured its real write speed for large files?
> Try
> something like:
>
> time dd if=/dev/zero of=/the/usb/drive count=1000 bs=1M
>
> You might be surprised.
>
> poc
Well, I am not surprised, but disappointed (seriously!!!)
$ time dd if=/dev/zero of=/run/media/jd/USB3_ST1-2K/test-wr-speed.txt
bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 57.6116 s, 18.2 MB/s
real 0m57.88s
user 0m0.00s
sys 0m1.28s
So much for 50MB/s write speed!!!!
USB flash drives only work efficiently at multiples of their internal
block size, which is not usually directly visible to the host (Google
for info on how flash drives actually work; hint: every write requires
the entire block to be zeroed and then overwritten). The claimed I/O
speed of these devices is achievable, just not under real conditions
with a generic driver.
poc
4:42 p.m.
On 07/13/2015 03:13 PM, Patrick O'Callaghan wrote:
On Mon, 2015-07-13 at 13:50 -0600, jd1008 wrote:
> I started a tar command from
> one external eSATA drive (ext4), connected to eSATA port,
> out to a USB flash drive (with vfat). The USB stick is touted
> to support 50MB/s write, 160MB/s read.
Have you actually measured its real write speed for large files? Try
something like:
time dd if=/dev/zero of=/the/usb/drive count=1000 bs=1M
You might be surprised.
poc
As I just replied with the actual read/write performance data
on this USB 3.0 SUperTalentExpress Drive (256GB), I am
much more surprised by the fact that while tarring a large
dir to it from a fast eSATA drive, the system becomes nearly
unusable!!!
So, if this "shtick" is so slow, why is writing to it killing the response
time for everything else?
6:15 p.m.
On Mon, 2015-07-13 at 15:42 -0600, jd1008 wrote:
On 07/13/2015 03:13 PM, Patrick O'Callaghan wrote:
> On Mon, 2015-07-13 at 13:50 -0600, jd1008 wrote:
> > I started a tar command from
> > one external eSATA drive (ext4), connected to eSATA port,
> > out to a USB flash drive (with vfat). The USB stick is touted
> > to support 50MB/s write, 160MB/s read.
> Have you actually measured its real write speed for large files?
> Try
> something like:
>
> time dd if=/dev/zero of=/the/usb/drive count=1000 bs=1M
>
> You might be surprised.
>
> poc
As I just replied with the actual read/write performance data
on this USB 3.0 SUperTalentExpress Drive (256GB), I am
much more surprised by the fact that while tarring a large
dir to it from a fast eSATA drive, the system becomes nearly
unusable!!!
So, if this "shtick" is so slow, why is writing to it killing the
response
time for everything else?
IIRC USB drives are not DMA driven, i.e. each filesystem block written
to the drive has to be pushed over the serial bus and handled by the
CPU. Also, check if the drive is mounted with the 'flush' option, which
sacrifices speed for greater integrity (i.e. less damage of the user
pulls the drive without ejecting it).
poc
7:38 p.m.
On 07/13/2015 05:15 PM, Patrick O'Callaghan wrote:
On Mon, 2015-07-13 at 15:42 -0600, jd1008 wrote:
> On 07/13/2015 03:13 PM, Patrick O'Callaghan wrote:
>> On Mon, 2015-07-13 at 13:50 -0600, jd1008 wrote:
>>> I started a tar command from
>>> one external eSATA drive (ext4), connected to eSATA port,
>>> out to a USB flash drive (with vfat). The USB stick is touted
>>> to support 50MB/s write, 160MB/s read.
>> Have you actually measured its real write speed for large files?
>> Try
>> something like:
>>
>> time dd if=/dev/zero of=/the/usb/drive count=1000 bs=1M
>>
>> You might be surprised.
>>
>> poc
> As I just replied with the actual read/write performance data
> on this USB 3.0 SUperTalentExpress Drive (256GB), I am
> much more surprised by the fact that while tarring a large
> dir to it from a fast eSATA drive, the system becomes nearly
> unusable!!!
> So, if this "shtick" is so slow, why is writing to it killing the
> response
> time for everything else?
IIRC USB drives are not DMA driven, i.e. each filesystem block written
to the drive has to be pushed over the serial bus and handled by the
CPU. Also, check if the drive is mounted with the 'flush' option, which
sacrifices speed for greater integrity (i.e. less damage of the user
pulls the drive without ejecting it).
poc
Good tip. I will check next time when I rerun the test.
7:52 p.m.
On 07/13/15 19:38, jd1008 wrote:
On 07/13/2015 05:15 PM, Patrick O'Callaghan wrote:
<<>>
> IIRC USB drives are not DMA driven, i.e. each filesystem block
written
> to the drive has to be pushed over the serial bus and handled by the
> CPU. Also, check if the drive is mounted with the 'flush' option, which
> sacrifices speed for greater integrity (i.e. less damage of the user
> pulls the drive without ejecting it).
>
> poc
Good tip. I will check next time when I rerun the test.
i agree with 'poc' 100% with both of his statements.
also, i think you might want to rethink your thinking.
you say you are connecting to a usb2 port and device is a usb3.
near all reading of spec for usb3 devices that will work on usb2
have a statement that thru put speeds will be less if used on usb2.
more for thought, is tar building finale file in memory, /tmp, or
on the usb3 drive?
--
peace out.
-+-
If Bill Gates got a dime for every time Windows crashes...
...oh, wait. He does. THAT explains it!
-+-
in a world with out fences, who needs gates.
-+-
CentOS GNU/Linux 6.6
tc,hago.
g
.
9:17 p.m.
On 07/13/2015 06:52 PM, g wrote:
more for thought, is tar building finale file in memory, /tmp, or on
the usb3 drive?
:)
Well, since I was using a pipe, I assume swap backed memory is used.
However, the un-tar process is running in parallel, so the RAM
pipe is shared between the producer "tar" and the consumer
tar, which populates the destination directory with the contents
of the source directory.
That said, I do not think that the entirety of the source dir is first
copied to the "pipe", before it is then consumed by the
"un"tar process. I think the RAM pipe is being read (emptied),
as the producer fills it (using thread synchronization primitives).
Just my guess. I have not looked at tar's source code.
9:43 p.m.
On 07/13/15 21:17, jd1008 wrote:
On 07/13/2015 06:52 PM, g wrote:
> more for thought, is tar building finale file in memory, /tmp, or on
> the usb3 drive?
:)
Well, since I was using a pipe, I assume swap backed memory is used.
However, the un-tar process is running in parallel, so the RAM
pipe is shared between the producer "tar" and the consumer
tar, which populates the destination directory with the contents
of the source directory.
That said, I do not think that the entirety of the source dir is first
copied to the "pipe", before it is then consumed by the
"un"tar process. I think the RAM pipe is being read (emptied),
as the producer fills it (using thread synchronization primitives).
Just my guess. I have not looked at tar's source code.
ok. you have now thrown in *pipe* and process *un-tar*.
are you using tar to pass files to target and have them end up as separate
files, or as a single archive file?
ria, all i see in your original post is "tar -C /sdc1 -xpvf" which would
indicate archiving to a single file.
just what is your actual command line?
--
peace out.
-+-
If Bill Gates got a dime for every time Windows crashes...
...oh, wait. He does. THAT explains it!
-+-
in a world with out fences, who needs gates.
-+-
CentOS GNU/Linux 6.6
tc,hago.
g
.
9:54 p.m.
On 07/13/2015 08:43 PM, g wrote:
On 07/13/15 21:17, jd1008 wrote:
> On 07/13/2015 06:52 PM, g wrote:
>> more for thought, is tar building finale file in memory, /tmp, or on
>> the usb3 drive?
> :)
> Well, since I was using a pipe, I assume swap backed memory is used.
> However, the un-tar process is running in parallel, so the RAM
> pipe is shared between the producer "tar" and the consumer
> tar, which populates the destination directory with the contents
> of the source directory.
> That said, I do not think that the entirety of the source dir is first
> copied to the "pipe", before it is then consumed by the
> "un"tar process. I think the RAM pipe is being read (emptied),
> as the producer fills it (using thread synchronization primitives).
> Just my guess. I have not looked at tar's source code.
>
>
ok. you have now thrown in *pipe* and process *un-tar*.
are you using tar to pass files to target and have them end up as separate
files, or as a single archive file?
ria, all i see in your original post is "tar -C /sdc1 -xpvf" which would
indicate archiving to a single file.
just what is your actual command line?
tar cf - BigDirArchive | tar -C /sdc1 -xpvf -
11:16 p.m.
On 07/13/15 21:54, jd1008 wrote:
On 07/13/2015 08:43 PM, g wrote:
<<<>>>
> ok. you have now thrown in *pipe* and process *un-tar*.
>
> are you using tar to pass files to target and have them end up as separate
> files, or as a single archive file?
>
> ria, all i see in your original post is "tar -C /sdc1 -xpvf" which would
> indicate archiving to a single file.
oops. -x = extract.
> just what is your actual command line?
>
>
tar cf - BigDirArchive | tar -C /sdc1 -xpvf -
ok. that does make for better understanding.
i stopped using tar years ago in favor of cpio, which has more advantages,
when i was using tape for archiving. works well with usb memory and usb
hdd drives. in fact, when i archive paths between hdd partitions or drives,
i use cpio instead of "cp -R".
to perform same using cpio, command line would be;
find path2BigDirArchive -depth -print0 | sort | cpio -p -adm /sdc1
or
find . -depth -print0 | sort | cpio -p -adm /sdc1
in above, "-p" is 'copy-pass mode', equivalent to "|" and
"-x" in tar.
because you are having response time issues with usb, i believe you will
find cpio to be more favorable than using tar.
--
peace out.
-+-
If Bill Gates got a dime for every time Windows crashes...
...oh, wait. He does. THAT explains it!
-+-
in a world with out fences, who needs gates.
-+-
CentOS GNU/Linux 6.6
tc,hago.
g
.
2:04 a.m.
double oops.
my thinking and typing were out of sync.
On 07/13/15 23:16, g wrote:
On 07/13/15 21:54, jd1008 wrote:
> On 07/13/2015 08:43 PM, g wrote:
<<<>>>
>> ok. you have now thrown in *pipe* and process *un-tar*.
>>
>> are you using tar to pass files to target and have them end up as separate
>> files, or as a single archive file?
>>
>> ria, all i see in your original post is "tar -C /sdc1 -xpvf" which
would
>> indicate archiving to a single file.
oops. -x = extract.
>> just what is your actual command line?
>>
>>
> tar cf - BigDirArchive | tar -C /sdc1 -xpvf -
>
should have read:
ok. that does make for better understanding.
as a suggestion;
i stopped using tar years ago in favor of cpio, which has more
advantages,
when i was using tape for archiving. works well with usb memory and usb
hdd drives. in fact, when i archive paths between hdd partitions or drives,
i use cpio instead of "cp -R".
to perform same using cpio, command line would be;
find path2BigDirArchive -depth -print0 | sort | cpio -p -adm /sdc1
or
find . -depth -print0 | sort | cpio -p -adm /sdc1
in above, "-p" is 'copy-pass mode', equivalent to "|" and
"-x" in tar.
because you are having response time issues with usb, i believe you will
find cpio to be more favorable than using tar.
--
peace out.
-+-
If Bill Gates got a dime for every time Windows crashes...
...oh, wait. He does. THAT explains it!
-+-
in a world with out fences, who needs gates.
-+-
CentOS GNU/Linux 6.6
tc,hago.
g
.
7:13 a.m.
On 14/07/15 07:42, jd1008 wrote:
On 07/13/2015 03:13 PM, Patrick O'Callaghan wrote:
> On Mon, 2015-07-13 at 13:50 -0600, jd1008 wrote:
>> I started a tar command from
>> one external eSATA drive (ext4), connected to eSATA port,
>> out to a USB flash drive (with vfat). The USB stick is touted
>> to support 50MB/s write, 160MB/s read.
> Have you actually measured its real write speed for large files? Try
> something like:
>
> time dd if=/dev/zero of=/the/usb/drive count=1000 bs=1M
>
> You might be surprised.
>
> poc
As I just replied with the actual read/write performance data
on this USB 3.0 SUperTalentExpress Drive (256GB), I am
much more surprised by the fact that while tarring a large
dir to it from a fast eSATA drive, the system becomes nearly
unusable!!!
So, if this "shtick" is so slow, why is writing to it killing the response
time for everything else?
I see the same sort of issue (F21, cinnamon desktop). If the disk write
or read is getting "backed up" with a queue of info, then that should
mean the desktop UI is more responsive, since the CPU is idle. But it
doesn't seem to work that way. It quite often seems to be waiting on a
large disk IO to complete.
On mine, possibly having once connected to a windows7 machine and copied
a mpg file from it to mine, but leaving the nemo window open might be a
trigger. And an hour after boot, the prelink operation runs (again,
should be set for tinniest priority, and rest of OS should be normal
especially given / is on an SSD (with modified time writing disabled).
Love to hear some things to try...
11:04 a.m.
On 07/14/2015 06:13 AM, David Timms wrote:
On 14/07/15 07:42, jd1008 wrote:
>
> On 07/13/2015 03:13 PM, Patrick O'Callaghan wrote:
>> On Mon, 2015-07-13 at 13:50 -0600, jd1008 wrote:
>>> I started a tar command from
>>> one external eSATA drive (ext4), connected to eSATA port,
>>> out to a USB flash drive (with vfat). The USB stick is touted
>>> to support 50MB/s write, 160MB/s read.
>> Have you actually measured its real write speed for large files? Try
>> something like:
>>
>> time dd if=/dev/zero of=/the/usb/drive count=1000 bs=1M
>>
>> You might be surprised.
>>
>> poc
> As I just replied with the actual read/write performance data
> on this USB 3.0 SUperTalentExpress Drive (256GB), I am
> much more surprised by the fact that while tarring a large
> dir to it from a fast eSATA drive, the system becomes nearly
> unusable!!!
> So, if this "shtick" is so slow, why is writing to it killing the response
> time for everything else?
I see the same sort of issue (F21, cinnamon desktop). If the disk write
or read is getting "backed up" with a queue of info, then that should
mean the desktop UI is more responsive, since the CPU is idle. But it
doesn't seem to work that way. It quite often seems to be waiting on a
large disk IO to complete.
On mine, possibly having once connected to a windows7 machine and copied
a mpg file from it to mine, but leaving the nemo window open might be a
trigger. And an hour after boot, the prelink operation runs (again,
should be set for tinniest priority, and rest of OS should be normal
especially given / is on an SSD (with modified time writing disabled).
Love to hear some things to try...
I wish I knew ...
I find that the desktop kernel and scheduler are tuned more
to server operations where FS buffers for all the threads are
given priority as the desktop is either non-existent or not in operatio.
memory is used mostly for these buffers (aka pagecache).
3236
days inactive
3237
days old
14 comments
4 participants
participants (4)
-
Development discussions related to Fedora -
g -
JD -
Patrick O'Callaghan