[announce] yum: parallel downloading
zpavlas at redhat.com
Mon May 21 10:33:51 UTC 2012
> The number of concurrent users is now lower because, well, each of
> them now completes a "yum update" in one third of the time.
I think Glen's concerns were that the consumed resources
(flow caches, TCP hash entries, sockets) may scale faster
than the aggregated downloading speed.
I am aware of this, and in most cases the downloader in urlgrabber
will make just 1 concurrent connection to a mirror, because:
1) The Nth concurrent connection to the same host is assumed
to be N times slower than 1st one, so we'll very likely
not select the same mirror again.
2) maxconnections=1 in every metalink I've seen so far.
This is a hard limit, we block until a download finishes
and reuse one connection when the limit is maxed out.
The reason for NOT banning >1 connections to the same host altogether
is that (as John Reiser wrote) 2nd connection does help quite a lot
when downloading many small files and just one mirror is available.
I agree that using strictly 1 connection and HTTP pipelining would
be even better, but we can't do that with libcurl.
More information about the devel