[announce] yum: parallel downloading

[Matt Domsch]

On Mon, May 21, 2012 at 05:33:51AM -0500, Zdenek Pavlas wrote:
> > The number of concurrent users is now lower because, well, each of
> > them now completes a "yum update" in one third of the time.
> 
> I think Glen's concerns were that the consumed resources
> (flow caches, TCP hash entries, sockets) may scale faster
> than the aggregated downloading speed.
> 
> I am aware of this, and in most cases the downloader in urlgrabber
> will make just 1 concurrent connection to a mirror, because:
> 
> 1) The Nth concurrent connection to the same host is assumed
>    to be N times slower than 1st one, so we'll very likely
>    not select the same mirror again.
> 
> 2) maxconnections=1 in every metalink I've seen so far.
>    This is a hard limit, we block until a download finishes
>    and reuse one connection when the limit is maxed out.

That's MirrorManager's doing.  I don't have a box for mirror admins to
tell MM otherwise - that they'd be happy with >1 connection.  I
suppose that could be added, default to 1.

> The reason for NOT banning >1 connections to the same host altogether
> is that (as John Reiser wrote) 2nd connection does help quite a lot
> when downloading many small files and just one mirror is available.
> I agree that using strictly 1 connection and HTTP pipelining would 
> be even better, but we can't do that with libcurl.

I'd be happy if yum/urlgrabber/libcurl finally used http keepalives.
Last time I looked (and it has been a while), it didn't, so you always
paid the TCP slow startup penalty for each package.

-- 
Matt Domsch
Technology Strategist
Dell | Office of the CTO