I/O or CPU bandwidth issue or wget issue or perhaps isp???

[Rick Stevens]

On 09/08/2012 10:07 AM, JD uttered this comment:
> fc16 (latest updates) + wget-1.12-4.fc16.i686, and system load
> negligible when I start wget.
> CPU is Athlon64 3700+ (but I run i686 kernel/apps).
>
> Sometimes, wget chokes
>
>
> 25% [+++++=====>                              ] 1,000,855,688 --.-K/s
> eta 27m 57s
>
> even though I can actually start another download in a a separate terminal)
> of the same file from the same server with no delays, which in turn after
> a while chokes as above. The download speeds fluctuate as low as a few
> hundred
> Kbytes/s  to 2.70 Mbytes/s. It usually chokes when it reaches the high
> end of
> download speeds.
>
> I have measured my disk i/o throughput using dd from one drive (on the
> internal
> eide bus) to the external eSATA drive connected to system via a dual
> port eSATA cardbus.
> The speed ranges from 20 to 27 megabytes/s.
>
> The wget download goes to the external eSATA drive /dev/sdb, on
> partition sdb5 (7200 rpm).
>
> So, I am at a total loss with the reason why this is happening (and
> frequently within the last few days).
>
> Also, during wget runs, iostat -chm 5
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            15.64    0.00   14.80   13.74    0.00   55.81
>
> Device:  tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
> sdb     7.82         0.00         1.94          0          9
> sdb1    0.63         0.00         0.01          0          0
> sdb2    0.00         0.00         0.00          0          0
> sdb3    2.54         0.00         0.04          0          0
> sdb4    0.00         0.00         0.00          0          0
> sdb5    4.65         0.00         1.89          0          8
>
> shows no obvious bottlenecks during 2.2 mbyte/s download speed snap shot.
>
> Does anyone have an idea what I should be monitoring in order to see
> the event that is leading to this?
> As I stated above, it does not seem to be the server because I can
> immediately start another transfer of same file from same server without
> seeing any delays.
> By the way, I am not ruling out the ISP as the cause of the hickup.
>
> Finally, ultimate reason I am posting this issue is because wget seems
> to remain
> in the chocked state and does not simply close the connection and
> immediately open
> a new one. Yesterday, it remained in the chocked state all night, until I
> interrupted wget and restarted the download (with -c of course) in the
> morning.

I've run into a similar choke scenario when using iptables. I haven't
bothered sorting out which of my rules causes the issue. When it
happens, I shut down iptables (I'm behind a hardware firewall anyway),
complete the download and turn iptables back on. Might be worth a look.
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital    ricks at alldigital.com -
- AIM/Skype: therps2        ICQ: 22643734            Yahoo: origrps2 -
-                                                                    -
-           Fear is finding a ".vbs" script in your Inbox            -
----------------------------------------------------------------------