Re: Public Mirror Listing request

[Adrian Reber]

On Mon, Jul 11, 2016 at 05:26:14PM -0400, James Hayden wrote:
>  Thanks. Added! How often are crawled sites re-indexed?
>  
>  Will continue adding the other locations now. Looking forward to contributing! 
So, you hit a bug in our crawler:

https://github.com/fedora-infra/mirrormanager2/issues/183

I added the corresponding HTTP URLs to your hosts, so that the crawler
is now running. Please add the RSYNC URLs as soon as possible as HTTP
crawling is really ineffective.

You do not need to create a Site for each Host. You can add all Hosts
under one Site. But you can structure it any way you want.

The crawler runs twice a day, but as is could not handle a mirror with
only a HTTPS URL (see issue above) it disabled your mirror after two
days.

Once the mirror is crawled it takes about an hour to re-create the
mirrorlist which is used by the clients. So in few hours you should see
the first clients connecting to your mirror.

Your mirror setup will be interesting as a our crawler will always hit
your LA node, so your CDN and our setup do not make much sense in
combination and crawling all your different URLs will always hit the LA
node. This means that the crawler will crawl multiple times the same
mirror. Let's see how this goes.

Thanks anyway for all your work getting this going for Fedora!

		Adrian

Public Mirror Listing request

[Adrian Reber]

On Tue, Jul 12, 2016 at 08:00:44AM +0200, Adrian Reber wrote:
>  On Mon, Jul 11, 2016 at 05:26:14PM -0400, James Hayden wrote:
>  > Thanks. Added! How often are crawled sites re-indexed?
>  > 
>  > Will continue adding the other locations now. Looking forward to contributing!
>  
>  So, you hit a bug in our crawler:
>  
>  https://github.com/fedora-infra/mirrormanager2/issues/183
>  
>  I added the corresponding HTTP URLs to your hosts, so that the crawler
>  is now running. Please add the RSYNC URLs as soon as possible as HTTP
>  crawling is really ineffective.
>  
>  You do not need to create a Site for each Host. You can add all Hosts
>  under one Site. But you can structure it any way you want.
>  
>  The crawler runs twice a day, but as is could not handle a mirror with
>  only a HTTPS URL (see issue above) it disabled your mirror after two
>  days.
>  
>  Once the mirror is crawled it takes about an hour to re-create the
>  mirrorlist which is used by the clients. So in few hours you should see
>  the first clients connecting to your mirror.
>  
>  Your mirror setup will be interesting as a our crawler will always hit
>  your LA node, so your CDN and our setup do not make much sense in
>  combination and crawling all your different URLs will always hit the LA
>  node. This means that the crawler will crawl multiple times the same
>  mirror. Let's see how this goes.
>  
>  Thanks anyway for all your work getting this going for Fedora! 
So, it is difficult. The IP of your mirror 89.45.92.5 resolves to

$ geoiplookup la-mirrors.evowise.com
GeoIP Country Edition: RO, Romania

$ geoiplookup ny-mirrors.evowise.com
GeoIP Country Edition: RO, Romania

It always resolves to Romania. So you really do not need to add anything
outside of Europe. If you are only providing CNAMEs and not the direct
IP to the nodes in LA or NY it doesn't make sense with Fedora's mirror
system. Looking at the output of a traceroute it seems I am always
redirected to Italy when testing from Europe and in the US it is
either LA or NY. So, I am sorry but this seems difficult to solve. We
rely on GeoIP and that does not work with your setup. It says Romania,
but it seems to be in Italy or somewhere else.


		Adrian

Public Mirror Listing request

[James Hayden]

Hi Adrian, 

Abuse contact for '89.45.92.0 - 89.45.92.255' is 'abuse(a)evowise.com' 
inetnum: 89.45.92.0 - 89.45.92.255 
netname: EvoWise-ANYCAST 
descr: Anycast Range 
country: US 
org: ORG-ES341-RIPE 
tech-c: TD4792-RIPE 
admin-c: TD4792-RIPE 

Can you please make an request to geoip (maxmind) to change this? 


Hayden 

From: "Adrian Reber" <adrian(a)lisas.de&gt; 
To: "James Hayden" <hayden(a)evowise.com&gt;, "mirror-admin"
<mirror-admin(a)fedoraproject.org&gt; 
Sent: Tuesday, July 12, 2016 4:52:19 AM 
Subject: Re: Public Mirror Listing request 

On Tue, Jul 12, 2016 at 08:00:44AM +0200, Adrian Reber wrote: 


On Mon, Jul 11, 2016 at 05:26:14PM -0400, James Hayden wrote: 
>  Thanks. Added! How often are crawled sites re-indexed? 
>  
>  Will continue adding the other locations now. Looking forward to contributing!  
So, you hit a bug in our crawler: 

https://github.com/fedora-infra/mirrormanager2/issues/183 

I added the corresponding HTTP URLs to your hosts, so that the crawler 
is now running. Please add the RSYNC URLs as soon as possible as HTTP 
crawling is really ineffective. 

You do not need to create a Site for each Host. You can add all Hosts 
under one Site. But you can structure it any way you want. 

The crawler runs twice a day, but as is could not handle a mirror with 
only a HTTPS URL (see issue above) it disabled your mirror after two 
days. 

Once the mirror is crawled it takes about an hour to re-create the 
mirrorlist which is used by the clients. So in few hours you should see 
the first clients connecting to your mirror. 

Your mirror setup will be interesting as a our crawler will always hit 
your LA node, so your CDN and our setup do not make much sense in 
combination and crawling all your different URLs will always hit the LA 
node. This means that the crawler will crawl multiple times the same 
mirror. Let's see how this goes. 

Thanks anyway for all your work getting this going for Fedora! 




So, it is difficult. The IP of your mirror 89.45.92.5 resolves to 

$ geoiplookup la-mirrors.evowise.com 
GeoIP Country Edition: RO, Romania 

$ geoiplookup ny-mirrors.evowise.com 
GeoIP Country Edition: RO, Romania 

It always resolves to Romania. So you really do not need to add anything 
outside of Europe. If you are only providing CNAMEs and not the direct 
IP to the nodes in LA or NY it doesn't make sense with Fedora's mirror 
system. Looking at the output of a traceroute it seems I am always 
redirected to Italy when testing from Europe and in the US it is 
either LA or NY. So, I am sorry but this seems difficult to solve. We 
rely on GeoIP and that does not work with your setup. It says Romania, 
but it seems to be in Italy or somewhere else. 


Adrian

Public Mirror Listing request

[James Hayden]

I've made the request to Maxmind here:
https://support.maxmind.com/geoip-data-correction-request/

The result...
"Thank you for sending your data correction. Location corrections are made on
Tuesdays."

----- Original Message -----
From: "Adrian Reber" <adrian(a)lisas.de&gt;
To: "James Hayden" <hayden(a)evowise.com&gt;, "mirror-admin"
<mirror-admin(a)fedoraproject.org&gt;
Sent: Tuesday, July 12, 2016 4:52:19 AM
Subject: Re: Public Mirror Listing request

On Tue, Jul 12, 2016 at 08:00:44AM +0200, Adrian Reber wrote:
>  On Mon, Jul 11, 2016 at 05:26:14PM -0400, James Hayden wrote:
>  > Thanks. Added! How often are crawled sites re-indexed?
>  > 
>  > Will continue adding the other locations now. Looking forward to contributing!
>  
>  So, you hit a bug in our crawler:
>  
>  https://github.com/fedora-infra/mirrormanager2/issues/183
>  
>  I added the corresponding HTTP URLs to your hosts, so that the crawler
>  is now running. Please add the RSYNC URLs as soon as possible as HTTP
>  crawling is really ineffective.
>  
>  You do not need to create a Site for each Host. You can add all Hosts
>  under one Site. But you can structure it any way you want.
>  
>  The crawler runs twice a day, but as is could not handle a mirror with
>  only a HTTPS URL (see issue above) it disabled your mirror after two
>  days.
>  
>  Once the mirror is crawled it takes about an hour to re-create the
>  mirrorlist which is used by the clients. So in few hours you should see
>  the first clients connecting to your mirror.
>  
>  Your mirror setup will be interesting as a our crawler will always hit
>  your LA node, so your CDN and our setup do not make much sense in
>  combination and crawling all your different URLs will always hit the LA
>  node. This means that the crawler will crawl multiple times the same
>  mirror. Let's see how this goes.
>  
>  Thanks anyway for all your work getting this going for Fedora! 
So, it is difficult. The IP of your mirror 89.45.92.5 resolves to

$ geoiplookup la-mirrors.evowise.com
GeoIP Country Edition: RO, Romania

$ geoiplookup ny-mirrors.evowise.com
GeoIP Country Edition: RO, Romania

It always resolves to Romania. So you really do not need to add anything
outside of Europe. If you are only providing CNAMEs and not the direct
IP to the nodes in LA or NY it doesn't make sense with Fedora's mirror
system. Looking at the output of a traceroute it seems I am always
redirected to Italy when testing from Europe and in the US it is
either LA or NY. So, I am sorry but this seems difficult to solve. We
rely on GeoIP and that does not work with your setup. It says Romania,
but it seems to be in Italy or somewhere else.


		Adrian

Public Mirror Listing request

[James Hayden]

Submitted all locations but only one shows up in mirror list. https bug? All mirrors are
available via both https and http.


Site name	Mirror	Admin Active	User Active	Private	Bandwith	Last crawled
Evowise.com Italy	it-mirrors.evowise.com	True	False	False	10000	2016-07-17
17:06:26.922988
Evowise.com Romania	ro-mirrors.evowise.com	True	False	False	10000	2016-07-17
17:06:36.529761
Evowise.com Spain	es-mirrors.evowise.com	True	False	False	10000	2016-07-17
17:07:17.553932
Evowise.com UK	        uk-mirrors.evowise.com	True	False	False	10000	2016-07-17
17:06:31.909716
Evowise.com US East	ny-mirrors.evowise.com	True	True	False	10000	2016-07-18
20:30:56.596761
Evowise.com US West	la-mirrors.evowise.com	True	False	False	10000	2016-07-17
17:05:12.645460

Can you update list?

Thanks,

Hayden James


----- Original Message -----
From: "Adrian Reber" <adrian(a)lisas.de&gt;
To: "James Hayden" <hayden(a)evowise.com&gt;, "mirror-admin"
<mirror-admin(a)fedoraproject.org&gt;
Sent: Tuesday, July 12, 2016 4:52:19 AM
Subject: Re: Public Mirror Listing request

On Tue, Jul 12, 2016 at 08:00:44AM +0200, Adrian Reber wrote:
>  On Mon, Jul 11, 2016 at 05:26:14PM -0400, James Hayden wrote:
>  > Thanks. Added! How often are crawled sites re-indexed?
>  > 
>  > Will continue adding the other locations now. Looking forward to contributing!
>  
>  So, you hit a bug in our crawler:
>  
>  https://github.com/fedora-infra/mirrormanager2/issues/183
>  
>  I added the corresponding HTTP URLs to your hosts, so that the crawler
>  is now running. Please add the RSYNC URLs as soon as possible as HTTP
>  crawling is really ineffective.
>  
>  You do not need to create a Site for each Host. You can add all Hosts
>  under one Site. But you can structure it any way you want.
>  
>  The crawler runs twice a day, but as is could not handle a mirror with
>  only a HTTPS URL (see issue above) it disabled your mirror after two
>  days.
>  
>  Once the mirror is crawled it takes about an hour to re-create the
>  mirrorlist which is used by the clients. So in few hours you should see
>  the first clients connecting to your mirror.
>  
>  Your mirror setup will be interesting as a our crawler will always hit
>  your LA node, so your CDN and our setup do not make much sense in
>  combination and crawling all your different URLs will always hit the LA
>  node. This means that the crawler will crawl multiple times the same
>  mirror. Let's see how this goes.
>  
>  Thanks anyway for all your work getting this going for Fedora! 
So, it is difficult. The IP of your mirror 89.45.92.5 resolves to

$ geoiplookup la-mirrors.evowise.com
GeoIP Country Edition: RO, Romania

$ geoiplookup ny-mirrors.evowise.com
GeoIP Country Edition: RO, Romania

It always resolves to Romania. So you really do not need to add anything
outside of Europe. If you are only providing CNAMEs and not the direct
IP to the nodes in LA or NY it doesn't make sense with Fedora's mirror
system. Looking at the output of a traceroute it seems I am always
redirected to Italy when testing from Europe and in the US it is
either LA or NY. So, I am sorry but this seems difficult to solve. We
rely on GeoIP and that does not work with your setup. It says Romania,
but it seems to be in Italy or somewhere else.


		Adrian

Public Mirror Listing request

[James Hayden]

I've also added the 'http' URLs to all the mirrors under "My Sites".
But still don't see our mirrors listed. Is there something I missed? 

Or still related to the https bug?:
https://github.com/fedora-infra/mirrormanager2/issues/183

Thanks

----- Original Message -----
From: "Adrian Reber" <adrian(a)lisas.de&gt;
To: "James Hayden" <hayden(a)evowise.com&gt;, "mirror-admin"
<mirror-admin(a)fedoraproject.org&gt;
Sent: Tuesday, July 12, 2016 4:52:19 AM
Subject: Re: Public Mirror Listing request

On Tue, Jul 12, 2016 at 08:00:44AM +0200, Adrian Reber wrote:
>  On Mon, Jul 11, 2016 at 05:26:14PM -0400, James Hayden wrote:
>  > Thanks. Added! How often are crawled sites re-indexed?
>  > 
>  > Will continue adding the other locations now. Looking forward to contributing!
>  
>  So, you hit a bug in our crawler:
>  
>  https://github.com/fedora-infra/mirrormanager2/issues/183
>  
>  I added the corresponding HTTP URLs to your hosts, so that the crawler
>  is now running. Please add the RSYNC URLs as soon as possible as HTTP
>  crawling is really ineffective.
>  
>  You do not need to create a Site for each Host. You can add all Hosts
>  under one Site. But you can structure it any way you want.
>  
>  The crawler runs twice a day, but as is could not handle a mirror with
>  only a HTTPS URL (see issue above) it disabled your mirror after two
>  days.
>  
>  Once the mirror is crawled it takes about an hour to re-create the
>  mirrorlist which is used by the clients. So in few hours you should see
>  the first clients connecting to your mirror.
>  
>  Your mirror setup will be interesting as a our crawler will always hit
>  your LA node, so your CDN and our setup do not make much sense in
>  combination and crawling all your different URLs will always hit the LA
>  node. This means that the crawler will crawl multiple times the same
>  mirror. Let's see how this goes.
>  
>  Thanks anyway for all your work getting this going for Fedora! 
So, it is difficult. The IP of your mirror 89.45.92.5 resolves to

$ geoiplookup la-mirrors.evowise.com
GeoIP Country Edition: RO, Romania

$ geoiplookup ny-mirrors.evowise.com
GeoIP Country Edition: RO, Romania

It always resolves to Romania. So you really do not need to add anything
outside of Europe. If you are only providing CNAMEs and not the direct
IP to the nodes in LA or NY it doesn't make sense with Fedora's mirror
system. Looking at the output of a traceroute it seems I am always
redirected to Italy when testing from Europe and in the US it is
either LA or NY. So, I am sorry but this seems difficult to solve. We
rely on GeoIP and that does not work with your setup. It says Romania,
but it seems to be in Italy or somewhere else.


		Adrian

Public Mirror Listing request

[Adrian Reber]

On Wed, Jul 20, 2016 at 05:41:39PM -0400, James Hayden wrote:
>  I've also added the 'http' URLs to all the mirrors under
> "My Sites". But still don't see our mirrors listed. Is there something I
> missed?  
There are still multiple problems. One is, that crawling your mirror is
really slow. It takes over three hours and that is why your mirror will
always be disabled. RSYNC would really help a lot.

The other problem is, that Fedora's MirrorManager cannot handle your
anycast setup. As already mentioned, our crawler will always crawl your
LA system and therefore all the other entries do not make sense. Please
delete all other Sites/Hosts. The only solution would be if you can
provide URLs which do not point to your anycast IP.

So either you can provide direct IP addresses of all your mirrors or
remove all other systems except the LA system from MirrorManager.

		Adrian

Public Mirror Listing request

[James Hayden]

Deleted. Thanks

----- Original Message -----
From: "Adrian Reber" <adrian(a)lisas.de&gt;
To: "James Hayden" <hayden(a)evowise.com&gt;
Cc: "mirror-admin" <mirror-admin(a)fedoraproject.org&gt;
Sent: Wednesday, July 20, 2016 11:16:52 PM
Subject: Re: Public Mirror Listing request

On Wed, Jul 20, 2016 at 05:41:39PM -0400, James Hayden wrote:
>  I've also added the 'http' URLs to all the mirrors under
> "My Sites". But still don't see our mirrors listed. Is there something I
> missed?  
There are still multiple problems. One is, that crawling your mirror is
really slow. It takes over three hours and that is why your mirror will
always be disabled. RSYNC would really help a lot.

The other problem is, that Fedora's MirrorManager cannot handle your
anycast setup. As already mentioned, our crawler will always crawl your
LA system and therefore all the other entries do not make sense. Please
delete all other Sites/Hosts. The only solution would be if you can
provide URLs which do not point to your anycast IP.

So either you can provide direct IP addresses of all your mirrors or
remove all other systems except the LA system from MirrorManager.

		Adrian