Fedora mirror mirrors.liquidweb.com

[Adrian Reber]

Hello mirrors.liquidweb.com admins,

Fedora's mirror crawler has trouble crawling your mirror as it does
return a unusual status code:

# curl http://mirrors.liquidweb.com/fedora-epel/ -v
* About to connect() to mirrors.liquidweb.com port 80 (#0)
*   Trying 69.167.187.144...
* Connected to mirrors.liquidweb.com (69.167.187.144) port 80 (#0)
> GET /fedora-epel/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: mirrors.liquidweb.com
> Accept: */*

< HTTP/1.1 000
< Server: nginx
< Date: Mon, 09 Oct 2017 12:37:13 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive

The crawler expects the status code 200 if everything works correctly and
ignores your mirror because of the status code 000. Any reason for
status code 000. It seems that the code 000 is something Akamai uses for
requests where the client aborted:

"""
Log Delivery Services (LDS) LDS will show a 000 for any 200 or 206
responses with a client abort: the object was served correctly from the
origin or edge, but the end-user terminated the connection/transaction
before it completed.
"""

Not sure if that is related to your situation. But right now your mirror
is disabled as it seems to be inaccessible (at least from the crawlers
point of view).

Adrian

[Mike Neir]

Greetings Adrian,

I'll follow up with my team and see if we can get to the bottom of this
issue. As far as I know, there shouldn't be any CDN involvement with our
mirror, just content served from one machine. Can you tell provide
information on which IP address(es) those queries might be coming from so
we can search more efficiently?

MN

Mike Neir
Infrastructure Administrator
Liquid Web, Inc.

On Mon, Oct 9, 2017 at 8:42 AM, Adrian Reber  wrote:
> Hello mirrors.liquidweb.com admins,
>
> Fedora's mirror crawler has trouble crawling your mirror as it does
> return a unusual status code:
>
> # curl http://mirrors.liquidweb.com/fedora-epel/ -v
> About to connect() to mirrors.liquidweb.com port 80 (#0)
> Trying 69.167.187.144...
> Connected to mirrors.liquidweb.com (69.167.187.144) port 80 (#0)
> > GET /fedora-epel/ HTTP/1.1
> > User-Agent: curl/7.29.0
> > Host: mirrors.liquidweb.com
> > Accept: */*
>
> < HTTP/1.1 000
> < Server: nginx
> < Date: Mon, 09 Oct 2017 12:37:13 GMT
> < Content-Type: text/html
> < Transfer-Encoding: chunked
> < Connection: keep-alive
>
> The crawler expects the status code 200 if everything works correctly and
> ignores your mirror because of the status code 000. Any reason for
> status code 000. It seems that the code 000 is something Akamai uses for
> requests where the client aborted:
>
> """
> Log Delivery Services (LDS) LDS will show a 000 for any 200 or 206
> responses with a client abort: the object was served correctly from the
> origin or edge, but the end-user terminated the connection/transaction
> before it completed.
> """
>
> Not sure if that is related to your situation. But right now your mirror
> is disabled as it seems to be inaccessible (at least from the crawlers
> point of view).
>             Adrian

[Adrian Reber]

Our crawlers should both connect natted from 209.132.181.102

Adrian

On Mon, Oct 09, 2017 at 10:31:30AM -0400, Mike Neir wrote:
> Greetings Adrian,
>
> I'll follow up with my team and see if we can get to the bottom of this
> issue. As far as I know, there shouldn't be any CDN involvement with our
> mirror, just content served from one machine. Can you tell provide
> information on which IP address(es) those queries might be coming from so
> we can search more efficiently?
>
> MN
>
> Mike Neir
> Infrastructure Administrator
> Liquid Web, Inc.
>
> On Mon, Oct 9, 2017 at 8:42 AM, Adrian Reber  wrote:
> > Hello mirrors.liquidweb.com admins,
> >
> > Fedora's mirror crawler has trouble crawling your mirror as it does
> > return a unusual status code:
> >
> > # curl http://mirrors.liquidweb.com/fedora-epel/ -v
> > About to connect() to mirrors.liquidweb.com port 80 (#0)
> > Trying 69.167.187.144...
> > Connected to mirrors.liquidweb.com (69.167.187.144) port 80 (#0)
> > > GET /fedora-epel/ HTTP/1.1
> > > User-Agent: curl/7.29.0
> > > Host: mirrors.liquidweb.com
> > > Accept: */*
> >
> > < HTTP/1.1 000
> > < Server: nginx
> > < Date: Mon, 09 Oct 2017 12:37:13 GMT
> > < Content-Type: text/html
> > < Transfer-Encoding: chunked
> > < Connection: keep-alive
> >
> > The crawler expects the status code 200 if everything works correctly and
> > ignores your mirror because of the status code 000. Any reason for
> > status code 000. It seems that the code 000 is something Akamai uses for
> > requests where the client aborted:
> >
> > """
> > Log Delivery Services (LDS) LDS will show a 000 for any 200 or 206
> > responses with a client abort: the object was served correctly from the
> > origin or edge, but the end-user terminated the connection/transaction
> > before it completed.
> > """
> >
> > Not sure if that is related to your situation. But right now your mirror
> > is disabled as it seems to be inaccessible (at least from the crawlers
> > point of view).
> >             Adrian

[Mike Neir]

Adrian,

We think we have things in a better state now. Can you try your checks
again and let us know if things look better?

MN

Mike Neir
Infrastructure Administrator
Liquid Web, Inc.
1-800-580-4985 x1365

On Mon, Oct 9, 2017 at 10:40 AM, Adrian Reber  wrote:
> Our crawlers should both connect natted from 209.132.181.102
>             Adrian
>
> On Mon, Oct 09, 2017 at 10:31:30AM -0400, Mike Neir wrote:
> > Greetings Adrian,
> >
> > I'll follow up with my team and see if we can get to the bottom of this
> > issue. As far as I know, there shouldn't be any CDN involvement with our
> > mirror, just content served from one machine. Can you tell provide
> > information on which IP address(es) those queries might be coming from so
> > we can search more efficiently?
> >
> > MN
> >
> > Mike Neir
> > Infrastructure Administrator
> > Liquid Web, Inc.
> >
> > On Mon, Oct 9, 2017 at 8:42 AM, Adrian Reber  wrote:
> > > Hello mirrors.liquidweb.com admins,
> > >
> > > Fedora's mirror crawler has trouble crawling your mirror as it does
> > > return a unusual status code:
> > >
> > > # curl http://mirrors.liquidweb.com/fedora-epel/ -v
> > > About to connect() to mirrors.liquidweb.com port 80 (#0)
> > > Trying 69.167.187.144...
> > > Connected to mirrors.liquidweb.com (69.167.187.144) port 80 (#0)
> > > > GET /fedora-epel/ HTTP/1.1
> > > > User-Agent: curl/7.29.0
> > > > Host: mirrors.liquidweb.com
> > > > Accept: */*
> > >
> > > < HTTP/1.1 000
> > > < Server: nginx
> > > < Date: Mon, 09 Oct 2017 12:37:13 GMT
> > > < Content-Type: text/html
> > > < Transfer-Encoding: chunked
> > > < Connection: keep-alive
> > >
> > > The crawler expects the status code 200 if everything works correctly
>
> and
> > > ignores your mirror because of the status code 000. Any reason for
> > > status code 000. It seems that the code 000 is something Akamai uses
>
> for
> > > requests where the client aborted:
> > >
> > > """
> > > Log Delivery Services (LDS) LDS will show a 000 for any 200 or 206
> > > responses with a client abort: the object was served correctly from the
> > > origin or edge, but the end-user terminated the connection/transaction
> > > before it completed.
> > > """
> > >
> > > Not sure if that is related to your situation. But right now your
>
> mirror
> > > is disabled as it seems to be inaccessible (at least from the crawlers
> > > point of view).
> > >             Adrian

[Adrian Reber]

Now everything looks prefect. Thanks for fixing.

In an hour or so you should start seeing clients being re-directed to
your mirror again.

Adrian

On Tue, Oct 10, 2017 at 10:31:36AM -0400, Mike Neir wrote:
> Adrian,
>
> We think we have things in a better state now. Can you try your checks
> again and let us know if things look better?
>
> MN
>
> Mike Neir
> Infrastructure Administrator
> Liquid Web, Inc.
> 1-800-580-4985 x1365
>
> On Mon, Oct 9, 2017 at 10:40 AM, Adrian Reber  wrote:
> > Our crawlers should both connect natted from 209.132.181.102
> >             Adrian
> >
> > On Mon, Oct 09, 2017 at 10:31:30AM -0400, Mike Neir wrote:
> > > Greetings Adrian,
> > >
> > > I'll follow up with my team and see if we can get to the bottom of this
> > > issue. As far as I know, there shouldn't be any CDN involvement with our
> > > mirror, just content served from one machine. Can you tell provide
> > > information on which IP address(es) those queries might be coming from so
> > > we can search more efficiently?
> > >
> > > MN
> > >
> > > Mike Neir
> > > Infrastructure Administrator
> > > Liquid Web, Inc.
> > >
> > > On Mon, Oct 9, 2017 at 8:42 AM, Adrian Reber  wrote:
> > > > Hello mirrors.liquidweb.com admins,
> > > >
> > > > Fedora's mirror crawler has trouble crawling your mirror as it does
> > > > return a unusual status code:
> > > >
> > > > # curl http://mirrors.liquidweb.com/fedora-epel/ -v
> > > > About to connect() to mirrors.liquidweb.com port 80 (#0)
> > > > Trying 69.167.187.144...
> > > > Connected to mirrors.liquidweb.com (69.167.187.144) port 80 (#0)
> > > > > GET /fedora-epel/ HTTP/1.1
> > > > > User-Agent: curl/7.29.0
> > > > > Host: mirrors.liquidweb.com
> > > > > Accept: */*
> > > >
> > > > < HTTP/1.1 000
> > > > < Server: nginx
> > > > < Date: Mon, 09 Oct 2017 12:37:13 GMT
> > > > < Content-Type: text/html
> > > > < Transfer-Encoding: chunked
> > > > < Connection: keep-alive
> > > >
> > > > The crawler expects the status code 200 if everything works correctly
> >
> > and
> > > > ignores your mirror because of the status code 000. Any reason for
> > > > status code 000. It seems that the code 000 is something Akamai uses
> >
> > for
> > > > requests where the client aborted:
> > > >
> > > > """
> > > > Log Delivery Services (LDS) LDS will show a 000 for any 200 or 206
> > > > responses with a client abort: the object was served correctly from the
> > > > origin or edge, but the end-user terminated the connection/transaction
> > > > before it completed.
> > > > """
> > > >
> > > > Not sure if that is related to your situation. But right now your
> >
> > mirror
> > > > is disabled as it seems to be inaccessible (at least from the crawlers
> > > > point of view).
> > > >             Adrian

[Mike Neir]

Excellent. Just for future reference on your side if you see it happening
for others, the problem was caused by the Nginx FancyIndexes module, which
we were using to insert company branding into directory indexes. Shutting
it off corrected the issue.

MN

Mike Neir
Senior DevOps Engineer
TechOps Team
Liquid Web, Inc.
1-800-580-4985

On Tue, Oct 10, 2017 at 11:03 AM, Adrian Reber  wrote:
> Now everything looks prefect. Thanks for fixing.
>
> In an hour or so you should start seeing clients being re-directed to
> your mirror again.
>             Adrian
>
> On Tue, Oct 10, 2017 at 10:31:36AM -0400, Mike Neir wrote:
> > Adrian,
> >
> > We think we have things in a better state now. Can you try your checks
> > again and let us know if things look better?
> >
> > MN
> >
> > Mike Neir
> > Infrastructure Administrator
> > Liquid Web, Inc.
> > 1-800-580-4985 x1365
> >
> > On Mon, Oct 9, 2017 at 10:40 AM, Adrian Reber  wrote:
> > > Our crawlers should both connect natted from 209.132.181.102
> > >             Adrian
> > >
> > > On Mon, Oct 09, 2017 at 10:31:30AM -0400, Mike Neir wrote:
> > > > Greetings Adrian,
> > > >
> > > > I'll follow up with my team and see if we can get to the bottom of
>
> this
> > > > issue. As far as I know, there shouldn't be any CDN involvement with
>
> our
> > > > mirror, just content served from one machine. Can you tell provide
> > > > information on which IP address(es) those queries might be coming
>
> from so
> > > > we can search more efficiently?
> > > >
> > > > MN
> > > >
> > > > Mike Neir
> > > > Infrastructure Administrator
> > > > Liquid Web, Inc.
> > > >
> > > > On Mon, Oct 9, 2017 at 8:42 AM, Adrian Reber
>
> wrote:
> > > > > Hello mirrors.liquidweb.com admins,
> > > > >
> > > > > Fedora's mirror crawler has trouble crawling your mirror as it does
> > > > > return a unusual status code:
> > > > >
> > > > > # curl http://mirrors.liquidweb.com/fedora-epel/ -v
> > > > > About to connect() to mirrors.liquidweb.com port 80 (#0)
> > > > > Trying 69.167.187.144...
> > > > > Connected to mirrors.liquidweb.com (69.167.187.144) port 80 (#0)
> > > > > > GET /fedora-epel/ HTTP/1.1
> > > > > > User-Agent: curl/7.29.0
> > > > > > Host: mirrors.liquidweb.com
> > > > > > Accept: */*
> > > > >
> > > > > < HTTP/1.1 000
> > > > > < Server: nginx
> > > > > < Date: Mon, 09 Oct 2017 12:37:13 GMT
> > > > > < Content-Type: text/html
> > > > > < Transfer-Encoding: chunked
> > > > > < Connection: keep-alive
> > > > >
> > > > > The crawler expects the status code 200 if everything works
>
> correctly
> > > and
> > > > > ignores your mirror because of the status code 000. Any reason for
> > > > > status code 000. It seems that the code 000 is something Akamai
>
> uses
> > > for
> > > > > requests where the client aborted:
> > > > >
> > > > > """
> > > > > Log Delivery Services (LDS) LDS will show a 000 for any 200 or 206
> > > > > responses with a client abort: the object was served correctly
>
> from the
> > > > > origin or edge, but the end-user terminated the
>
> connection/transaction
> > > > > before it completed.
> > > > > """
> > > > >
> > > > > Not sure if that is related to your situation. But right now your
> > >
> > > mirror
> > > > > is disabled as it seems to be inaccessible (at least from the
>
> crawlers
> > > > > point of view).
> > > > >             Adrian