[announce] yum: parallel downloading
Matt_Domsch at dell.com
Tue May 22 13:36:17 UTC 2012
On Mon, May 21, 2012 at 05:33:51AM -0500, Zdenek Pavlas wrote:
> > The number of concurrent users is now lower because, well, each of
> > them now completes a "yum update" in one third of the time.
> I think Glen's concerns were that the consumed resources
> (flow caches, TCP hash entries, sockets) may scale faster
> than the aggregated downloading speed.
> I am aware of this, and in most cases the downloader in urlgrabber
> will make just 1 concurrent connection to a mirror, because:
> 1) The Nth concurrent connection to the same host is assumed
> to be N times slower than 1st one, so we'll very likely
> not select the same mirror again.
> 2) maxconnections=1 in every metalink I've seen so far.
> This is a hard limit, we block until a download finishes
> and reuse one connection when the limit is maxed out.
That's MirrorManager's doing. I don't have a box for mirror admins to
tell MM otherwise - that they'd be happy with >1 connection. I
suppose that could be added, default to 1.
> The reason for NOT banning >1 connections to the same host altogether
> is that (as John Reiser wrote) 2nd connection does help quite a lot
> when downloading many small files and just one mirror is available.
> I agree that using strictly 1 connection and HTTP pipelining would
> be even better, but we can't do that with libcurl.
I'd be happy if yum/urlgrabber/libcurl finally used http keepalives.
Last time I looked (and it has been a while), it didn't, so you always
paid the TCP slow startup penalty for each package.
Dell | Office of the CTO
More information about the devel