Yum 2.2 w/ HTTP proxy fails w/ error "Cannot find a valid baseurl for repo:"

Fri Apr 1 06:12:54 UTC 2005

On Thu, 2005-03-31 at 23:21 +0100, Mike Fleetwood wrote:
> On Wed, 30 Mar 2005, Paul Howarth wrote:
> 
> > On Wed, 2005-03-30 at 12:46 -0800, Kam Leo wrote:
> >> On Wed, 30 Mar 2005 08:40:09 +0100, Paul Howarth <paul at city-fan.org> wrote:
> >>> On Tue, 2005-03-29 at 22:35 +0100, Mike Fleetwood wrote:
> >>>> Recently "yum check-update" and "yum update" has started producing the
> >>>> error "Cannot find a valid baseurl for repo: ...".  Googling suggests
> >>>> that other occurrences have been caused by network connectivity errors
> >>>> or miss configured yum repository locations.  I am convinced that in my
> >>>> case it is yum 2.2's inability to successfully download files through a
> >>>> HTTP proxy that is the problem.  To work around the problem I
> >>>> reinstalled yum 2.1 and upgrade using "yum --exclude=yum update".
> >>>
> >>> yum 2.2 provides a way to specify the proxy globally or separately for
> >>> each repo if required (see "man yum.conf"). Did you try that?
> >>
> >> The OP commented out the baseurl line in
> >> /et/cyum.repos.d/fedora-updates.repo leaving only mirrorlist as the
> >> repositoryid.  Per 'man yum.conf':
> >
> > This is perfectly normal. The baseurl is in fact commented out by
> > default. See:
> > http://www.mirrorservice.org/sites/download.fedora.redhat.com/pub/fedora/linux/core/3/i386/os/Fedora/RPMS/fedora-release-3-8.i386.rpm//etc/yum.repos.d/fedora-updates.repo?extract=true
> >
> >> Either the mirrorlist list is defective or there is a bug in yum-2.2.
> >
> > Clearly the proxy configuration using environment variables, which used
> > to work, is no longer doing so. I would however be interested to hear if
> > the yum.conf way of specifying proxies now works in yum 2.2.
> 
> Executive summary:
> I was partially wrong.  Yum only doesn't work properly with HTTP
> proxies requiring authorisation.  Yum 2.1 copes, 2.2 doesn't.  Not
> using mirrorlist _might_ work around it.
> 
> 
> Details:
> I have been doing lots of packet tracing, discovery bad assumptions
> I have made, and found something closer to my actual problem with yum.
> 
> My initial problem was with yum 2.2 on FC3 at work where I have to use
> a Microsoft ISA web proxy with password authentication.  Yum 2.1 often
> reported errors left, right and centre but would eventually succeed in
> downloading the required RPMs.  When I upgraded to yum 2.2 it never
> succeeded and still doesn't.  More on this later ...
> 
> Flash back to home were I did some brief testing on FC3 which is behind
> Stone (http://www.gcd.org/sengoku/stone/) web proxy.  It must have
> failed for network problems a couple of times and I assumed that yum
> 2.2 was the problem.  Sorry for that.  Yum 2.1 and 2.2 successfully
> work though this web proxy which doesn't require authentication.
> 
> Packet tracing of yum 2.1 and 2.2 using HTTP_PROXY environment variable
> to define the Stone web proxy, which doesn't require authentication,
> shows that yum works OK.  Yum 2.1 actually downloads the mirror list
> for all repos even if they aren't enabled.  It does only download
> repomd.xml for enable repos though.  Yum 2.2 only gets the mirror list
> for repos that are actually enabled.  An improvement.
> 
> Now on to packet tracing of yum 2.1 and 2.2 at work through my
> companies MS ISA web proxy which requires authentication ...
> I have found that yum doesn't send the HTTP headers correctly when yum
> is using an authenticating web proxy.  This is when defining the web
> proxy using something like:
>    export HTTP_PROXY=http://username:password@webproxy.example.com:8080/
> MS ISA web proxy offers to keep the TCP connection open so it can be
> used to send more HTTP request over, with the "Keep-Alive" header
> which yum tries to use.  A typical TCP conversation goes like this:
>    yum>    GET ...
>    yum>    Host: ...
>    yum>    Accept-Encoding: identity
>    yum>    Proxy-authorization: Basic xxxxxxxxxxxxxxxxx
>    yum>                      <--- Blank line which shouldn't be here
>    yum>    User-agent: urlgrabber/2.9.2
>    yum>
>    proxy<  HTTP/1.1 200 OK
>    proxy<  ...
>    proxy<  Content-Length: ...
>    proxy<  ...
>    proxy<  Keep-Alive: ...
>    proxy<
>    proxy<  [Requested URL contents ...]
>    proxy<  ...
>    proxy<  HTTP/1.1 400 Bad Request ( The data is invalid. )
>    proxy<  ...
>    proxy<  Connection: close
>    proxy<  Proxy-Connection: close
>    proxy<  ...
>    yum>    GET ...
>    yum>    Host: ...
>    yum>    Accept-Encoding: identity
>    yum>    Proxy-authorization: Basic xxxxxxxxxxxxxxxxx
>    yum>                      <--- Blank line which shouldn't be here
>    yum>    User-agent: urlgrabber/2.9.2
>    yum>
> And the connection closes or times out.
> 
> So Yum (or urlgrabber) is incorrectly inserting a new line after the
> Proxy-authorization header before adding the User-agent header.  MS
> ISA web proxy thinks that it's 2 separate requests, the first works and
> the second results in an error message.  Yum doesn't realise as it only
> reads the specified number of bytes for its first request.  It then
> reuses the connection, sending another request.  It reads the
> previously buffer / unread data and see the error from the proxy.  The
> difference between yum 2.1 and 2.2 is that the particular files which
> they try to read from closed sockets differ, probably because yum 2.1
> reads mirror lists for repos its not using and yum 2.2 doesn't.  Yum
> 2.1 only ever sees the failures on files it can and does re-try, so it
> eventually succeeds.  Where as as yum 2.2 sees the failures on getting
> the mirror lists which it doesn't re-try so always fails.
> 
> This explanation suggests not using mirrorlist parameter and switching
> back to baseurl instead as a possible work around for yum 2.2.  I will
> try this in a later.
> 
> With all this I have only done one or two tests with yum 2.2 specifying
> proxy parameters in the configuration file.  Results seem to indicate
> that it just ignores these but don't count on this being the truth.

Excellent detective work!

I'd suggest raising a bug on yum for this, and/or contacting the author,
Seth Vidal <skvidal at phy dot duke dot edu>, who I've always found to
be very responsive.

Paul.
-- 
Paul Howarth <paul at city-fan.org>