Need some help for mirrorlist in China

[Zamir Sun]

Hi Mirror Admin team,

Recently, when I am checking the mirror lists in China, I see there is
only  mirrors.njupt.edu.cn listed for the repo fedora-28[1]. And
Checking the corresponding Fedora 27 one, I only see
mirrors.tuna.tsinghua.edu.cn
mirrors.163.com
mirrors.njupt.edu.cn

For fedora-updates of Fedora 28[2], still only the above three mirror
listed. While for fedora-updates of Fedora 27, only the two listed
mirrors.163.com
mirrors.njupt.edu.cn

According to my knowledge, the three biggest mirrors in China are

mirrors.tuna.tsinghua.edu.cn[3] (mentioned as TUNA below)
mirrors.ustc.edu.cn[4] (mentioned as USTC below)
mirrors.163.com

However USTC is not listed in any of the above. And TUNA is listed
only on the Fedora-27 repo. Given both of them are pretty big and
still listed on mirror page[5], can you check why they are not listed
by metalink? Especially TUNA has the biggest bandwith and USTC has
connection to every ISP in China.

I am copying their staff list as well.


Thanks in advance for your help!

[1]
https://mirrors.fedoraproject.org/metalink?repo=fedora-28&arch=x86_64...
[2]
https://mirrors.fedoraproject.org/metalink?repo=updates-released-f28&...
[3] https://mirrors.tuna.tsinghua.edu.cn/fedora/
[4] http://mirrors.ustc.edu.cn/fedora/releases/28/
[5] https://admin.fedoraproject.org/mirrormanager/mirrors/Fedora/28

-- 
Ziqian SUN (Zamir)
zsun(a)fedoraproject.org
GPG : 1D86 6D4A 49CE 4BBD 72CF FCF5 D856 6E11 F2A0 525E
Want to know more about Fedora?
Visit https://fedoraproject.org/wiki/

[Adrian Reber]

On Wed, May 09, 2018 at 07:29:24PM +0800, Zamir Sun wrote:
>  Recently, when I am checking the mirror lists in China, I see there
> is
>  only  mirrors.njupt.edu.cn listed for the repo fedora-28[1]. And
>  Checking the corresponding Fedora 27 one, I only see
>  mirrors.tuna.tsinghua.edu.cn
>  mirrors.163.com
>  mirrors.njupt.edu.cn
>  
>  For fedora-updates of Fedora 28[2], still only the above three mirror
>  listed. While for fedora-updates of Fedora 27, only the two listed
>  mirrors.163.com
>  mirrors.njupt.edu.cn
>  
>  According to my knowledge, the three biggest mirrors in China are
>  
>  mirrors.tuna.tsinghua.edu.cn[3] (mentioned as TUNA below)
>  mirrors.ustc.edu.cn[4] (mentioned as USTC below)
>  mirrors.163.com
>  
>  However USTC is not listed in any of the above. And TUNA is listed
>  only on the Fedora-27 repo. Given both of them are pretty big and
>  still listed on mirror page[5], can you check why they are not listed
>  by metalink? Especially TUNA has the biggest bandwith and USTC has
>  connection to every ISP in China.
>  
>  I am copying their staff list as well. 
Thanks for having a closer look. Just also talked with a USTC mirror
admin on IRC.

USTC is wrongly configured in mirrormanager, basically all paths
configured are pointing to non existing directories on the mirror. They
specify https://mirrors.ustc.edu.cn/fedora/linux as the root URL for
their Fedora mirror which returns 404. The URL for EPEL is also wrong.

TUNA is mostly correct. The reason why they to not serve Fedora 28 is
that they do not mirror development and we are waiting some days/weeks
until we switch from development to release. This will happen soon and
TUNA should appear as a mirror.

The same is true for 163.com.

It also seems that the last run of our crawler had trouble reaching any
sites in China. This might have been only a temporary problem.

		Adrian

[Zamir SUN]

On 05/09/2018 08:26 PM, Adrian Reber wrote:
>  On Wed, May 09, 2018 at 07:29:24PM +0800, Zamir Sun wrote:
> > Recently, when I am checking the mirror lists in China, I see there is
> > only  mirrors.njupt.edu.cn listed for the repo fedora-28[1]. And
> > Checking the corresponding Fedora 27 one, I only see
> > mirrors.tuna.tsinghua.edu.cn
> > mirrors.163.com
> > mirrors.njupt.edu.cn
> >
> > For fedora-updates of Fedora 28[2], still only the above three mirror
> > listed. While for fedora-updates of Fedora 27, only the two listed
> > mirrors.163.com
> > mirrors.njupt.edu.cn
> >
> > According to my knowledge, the three biggest mirrors in China are
> >
> > mirrors.tuna.tsinghua.edu.cn[3] (mentioned as TUNA below)
> > mirrors.ustc.edu.cn[4] (mentioned as USTC below)
> > mirrors.163.com
> >
> > However USTC is not listed in any of the above. And TUNA is listed
> > only on the Fedora-27 repo. Given both of them are pretty big and
> > still listed on mirror page[5], can you check why they are not listed
> > by metalink? Especially TUNA has the biggest bandwith and USTC has
> > connection to every ISP in China.
> >
> > I am copying their staff list as well.
>  
>  Thanks for having a closer look. Just also talked with a USTC mirror
>  admin on IRC.
>  
>  USTC is wrongly configured in mirrormanager, basically all paths
>  configured are pointing to non existing directories on the mirror. They
>  specify https://mirrors.ustc.edu.cn/fedora/linux as the root URL for
>  their Fedora mirror which returns 404. The URL for EPEL is also wrong.
>   
Thanks for the updates.
I believe USTC mirror admin will fix this shortly.

>  TUNA is mostly correct. The reason why they to not serve Fedora 28
> is
>  that they do not mirror development and we are waiting some days/weeks
>  until we switch from development to release. This will happen soon and
>  TUNA should appear as a mirror.
>  
>  The same is true for 163.com.
>   
This makes sense. It's a known status that they do not mirror development.

Thanks again for the details.

>  It also seems that the last run of our crawler had trouble reaching
> any
>  sites in China. This might have been only a temporary problem.
>  
>  		Adrian
>  
>  
>  
>  _______________________________________________
>  Mirror-admin mailing list -- mirror-admin(a)lists.fedoraproject.org
>  To unsubscribe send an email to mirror-admin-leave(a)lists.fedoraproject.org
>   
-- 
Ziqian SUN (Zamir)
GPG : 1D86 6D4A 49CE 4BBD 72CF FCF5 D856 6E11 F2A0 525E
Want to know more about Fedora?
Visit https://fedoraproject.org/wiki/
Ready to contribute? See https://whatcanidoforfedora.org/

[Shengjing Zhu]

Hi Adrian,

On Wed, May 9, 2018 at 8:26 PM, Adrian Reber <adrian(a)lisas.de&gt; wrote:
>  Thanks for having a closer look. Just also talked with a USTC mirror
>  admin on IRC.
>
>  USTC is wrongly configured in mirrormanager, basically all paths
>  configured are pointing to non existing directories on the mirror. They
>  specify https://mirrors.ustc.edu.cn/fedora/linux as the root URL for
>  their Fedora mirror which returns 404. The URL for EPEL is also wrong.
>  
In last talk on IRC, this is not the root the cause of the problem.

Few days ago, the old mirror site was deleted. I also tried to use
quick-fedora-mirror to sync fedora-secondary and epel. The checkin was
successful several times.
But I still can't see USTC appeared in the mirror list. P.S. I tested
from university IP I configured in "Site-local Netblocks"

When I'm writing,
https://mirrors.fedoraproject.org/metalink?repo=fedora-27&arch=aarch6...
result is empty.
The FILTEREXP I configured for fedora-secondary is
'(/s390/|/s390x/|/ppc64le/|/ppc64/|/source|/SRPMS|/debug/|/releases/test|/updates/testing|/development/)',
I'm not sure if this matters.


-- 
Linux User Group
University of Science and Technology of China
Homepage: https://lug.ustc.edu.cn/
E-Mail: lug(a)ustc.edu.cn
Shengjing Zhu <zhsj(a)ustclug.org&gt;

[Adrian Reber]

On Sat, May 12, 2018 at 01:56:29PM +0800, Shengjing Zhu wrote:
>  On Wed, May 9, 2018 at 8:26 PM, Adrian Reber <adrian(a)lisas.de&gt;
> wrote:
>  >
>  > Thanks for having a closer look. Just also talked with a USTC mirror
>  > admin on IRC.
>  >
>  > USTC is wrongly configured in mirrormanager, basically all paths
>  > configured are pointing to non existing directories on the mirror. They
>  > specify https://mirrors.ustc.edu.cn/fedora/linux as the root URL for
>  > their Fedora mirror which returns 404. The URL for EPEL is also wrong.
>  >
>  
>  In last talk on IRC, this is not the root the cause of the problem.
>  
>  Few days ago, the old mirror site was deleted. I also tried to use
>  quick-fedora-mirror to sync fedora-secondary and epel. The checkin was
>  successful several times.
>  But I still can't see USTC appeared in the mirror list. P.S. I tested
>  from university IP I configured in "Site-local Netblocks"
>  
>  When I'm writing,
>
> https://mirrors.fedoraproject.org/metalink?repo=fedora-27&arch=aarch6...
>  result is empty.
>  The FILTEREXP I configured for fedora-secondary is
>
> '(/s390/|/s390x/|/ppc64le/|/ppc64/|/source|/SRPMS|/debug/|/releases/test|/updates/testing|/development/)',
>  I'm not sure if this matters. 
Not really sure what is happening. Currently I assume that your
report_mirror is not working correctly. Is FILTEREXP from
quick-fedora-mirror? Or are you using report_mirror the stand-alone
python tool. It would be interesting to see what the server answers when
you run report_mirror.

My current theory is, your report_mirror config is not correct and
deletes all directories listed as up to date from your mirror in
MirrorManager. Once the crawler scans your mirror it is updated
correctly again. So the state of the mirror flips between crawling and
report_mirror.

		Adrian

[Shengjing Zhu]

On Mon, May 14, 2018 at 1:56 AM, Adrian Reber <adrian(a)lisas.de&gt; wrote:
>  Not really sure what is happening. Currently I assume that your
>  report_mirror is not working correctly. Is FILTEREXP from
>  quick-fedora-mirror? Or are you using report_mirror the stand-alone
>  python tool. It would be interesting to see what the server answers when
>  you run report_mirror.
>  
FILTEREXP is from quick-fedora-mirror. I'm not using the stand alone
report_mirror. I'm using quick-fedora-mirror to checkin.
I didn't see this zsh version has useful log.
Anyway the log is attached at:
http://ftp.ustclug.org/~zsj/logs/fedora-secondary.log.txt

>  My current theory is, your report_mirror config is not correct and
>  deletes all directories listed as up to date from your mirror in
>  MirrorManager. Once the crawler scans your mirror it is updated
>  correctly again. So the state of the mirror flips between crawling and
>  report_mirror. 
I think when the check is finished,
the "Up-to-Date Directories this host carries" on
https://admin.fedoraproject.org/mirrormanager/host/2266/category/7937
is correct. Maybe the crawler is the reason the site is marked as not
update2date?


-- 
Linux User Group
University of Science and Technology of China
Homepage: https://lug.ustc.edu.cn/
E-Mail: lug(a)ustc.edu.cn
Shengjing Zhu <zhsj(a)ustclug.org&gt;

[Adrian Reber]

On Mon, May 14, 2018 at 02:10:53AM +0800, Shengjing Zhu wrote:
>  > My current theory is, your report_mirror config is not correct
> and
>  > deletes all directories listed as up to date from your mirror in
>  > MirrorManager. Once the crawler scans your mirror it is updated
>  > correctly again. So the state of the mirror flips between crawling and
>  > report_mirror.
>  
>  I think when the check is finished,
>  the "Up-to-Date Directories this host carries" on
>  https://admin.fedoraproject.org/mirrormanager/host/2266/category/7937
>  is correct. Maybe the crawler is the reason the site is marked as not
>  update2date? 
You are right, the crawler had a bug. With the recent change to category
based crawling a bug in the crawler was uncovered:

https://github.com/fedora-infra/mirrormanager2/issues/249

Thanks for pointing it out. This should be fixed now.

		Adrian