Hello
We would like to publish a new mirror in our country, Uruguay.
The ip address of our mirror is: 190.64.49.124
The mirror is located in: Montevideo, Uruguay
Outbound bandwidth is: 500Mbps
The institution is: Facultad de Derecho, Universidad de la República
Supported protocols and their URLs are:
http -> http://espejito.fder.edu.uy/fedora/
https -> https://espejito.fder.edu.uy/fedora/
rsync -> rsync://espejito.fder.edu.uy/fedora
quick-fedora-mirror script is run hourly.
best regards
ariel
PS: as far as I understood, there is no need for report-mirror when using quick-fedora-mirror. Please confirm so.
On Thu, Oct 25, 2018 at 03:02:52PM -0300, ariel sabiguero yawelak wrote:
We would like to publish a new mirror in our country, Uruguay.
The ip address of our mirror is: 190.64.49.124
The mirror is located in: Montevideo, Uruguay
Outbound bandwidth is: 500Mbps
The institution is: Facultad de Derecho, Universidad de la República
Supported protocols and their URLs are:
http -> http://espejito.fder.edu.uy/fedora/ https -> https://espejito.fder.edu.uy/fedora/ rsync -> rsync://espejito.fder.edu.uy/fedoraquick-fedora-mirror script is run hourly.
Please use a tier 1 mirror[1] close to you; create an account in the MirrorManager[2]; register your mirror in our MirrorManager; enter your local netblocks in the mirror manager; run report_mirror[3] (if possible) and as soon as you are listed in the database as up to date yum/dnf will be redirected to your mirror.
All this information can be found in more details in the wiki at:
https://fedoraproject.org/wiki/Infrastructure/Mirroring
Thanks for supporting Fedora. If you have any further questions do not hesitate to ask.
It would help us a lot if you could provide rsync access for our crawler. Our crawler checks the content of your mirror and using rsync requires only one network connection in contrast to crawling via HTTP which can take up one connection per directory/file on your mirror.
Adrian
[1] https://fedoraproject.org/wiki/Infrastructure/Mirroring/Tiering [2] https://fedoraproject.org/wiki/Infrastructure/MirrorManager [3] https://fedoraproject.org/wiki/Infrastructure/Mirroring
Thanks for your answer.
Where do I state the protocols we serve? We server rsync, https and http. I found no way to register that on MirrorManager interface
regards
ariel
Please use a tier 1 mirror[1] close to you; create an account in the MirrorManager[2]; register your mirror in our MirrorManager; enter your local netblocks in the mirror manager; run report_mirror[3] (if possible) and as soon as you are listed in the database as up to date yum/dnf will be redirected to your mirror.
All this information can be found in more details in the wiki at:
https://fedoraproject.org/wiki/Infrastructure/Mirroring
Thanks for supporting Fedora. If you have any further questions do not hesitate to ask.
It would help us a lot if you could provide rsync access for our crawler. Our crawler checks the content of your mirror and using rsync requires only one network connection in contrast to crawling via HTTP which can take up one connection per directory/file on your mirror.
Adrian[1] https://fedoraproject.org/wiki/Infrastructure/Mirroring/Tiering [2] https://fedoraproject.org/wiki/Infrastructure/MirrorManager [3] https://fedoraproject.org/wiki/Infrastructure/Mirroring
On Tue, Oct 30, 2018 at 09:46:34AM -0300, ariel sabiguero yawelak wrote:
Where do I state the protocols we serve? We server rsync, https and http. I found no way to register that on MirrorManager interface
For each category you carry (Fedora Linux, EPEL, Archive), you add the category under 'Categories Carried' and once that is there you can add URLs to the categories.
Adrian
Ok, seems that I am getting closer :-)
For rsync protocol, should I use a single specialize one for fedora and one for epel?
regards
ariel
El 30/10/18 a las 9:48, Adrian Reber escribió:
On Tue, Oct 30, 2018 at 09:46:34AM -0300, ariel sabiguero yawelak wrote:
Where do I state the protocols we serve? We server rsync, https and http. I found no way to register that on MirrorManager interface
For each category you carry (Fedora Linux, EPEL, Archive), you add the category under 'Categories Carried' and once that is there you can add URLs to the categories.
Adrian
On Tue, Oct 30, 2018 at 10:04:30AM -0300, ariel sabiguero yawelak wrote:
Ok, seems that I am getting closer :-)
For rsync protocol, should I use a single specialize one for fedora and one for epel?
You can use whichever URL you want. It just has to point to the top-level directory of the category. It has to point to the same place the HTTP URLs do.
Adrian
Still unable to mirror release 29 (sure, not published yet).
How do we get promoted? otherwise, we won't be able to help until it is released... and we get synchronized.
regards
ariel
El 30/10/18 a las 11:34, Adrian Reber escribió:
On Tue, Oct 30, 2018 at 10:04:30AM -0300, ariel sabiguero yawelak wrote:
Ok, seems that I am getting closer :-)
For rsync protocol, should I use a single specialize one for fedora and one for epel?
You can use whichever URL you want. It just has to point to the top-level directory of the category. It has to point to the same place the HTTP URLs do.
Adrian
Well, no errors, but if I go to:
http://mirrors.kernel.org/fedora/releases/
I can see the folder for release 29
When I check:
http://espejito.fder.edu.uy/fedora/fedora/linux/releases/
there is nothing like release 29
I am rsyncing the whole enchilada from REMOTE=rsync://mirrors.kernel.org
but I don't see the same
I hope my explanation is clearer now.
regards
ariel
El 30/10/18 a las 15:02, Adrian Reber escribió:
On Tue, Oct 30, 2018 at 02:37:27PM -0300, ariel sabiguero yawelak wrote:
Still unable to mirror release 29 (sure, not published yet).
What do you mean exactly? What error do you see?
Adrian
On Tue, Oct 30, 2018 at 03:34:33PM -0300, ariel sabiguero yawelak wrote:
Well, no errors, but if I go to:
http://mirrors.kernel.org/fedora/releases/
I can see the folder for release 29
When I check:
http://espejito.fder.edu.uy/fedora/fedora/linux/releases/
there is nothing like release 29
I am rsyncing the whole enchilada from REMOTE=rsync://mirrors.kernel.org
but I don't see the same
I hope my explanation is clearer now.
Yes it does. It seems mirrors.kernel.org has not synced yet all the files. Just try another tier 1 mirror.
Adrian
On Tue, Oct 30, 2018 at 03:34:58PM +0100, Adrian Reber wrote:
On Tue, Oct 30, 2018 at 10:04:30AM -0300, ariel sabiguero yawelak wrote:
Ok, seems that I am getting closer :-)
For rsync protocol, should I use a single specialize one for fedora and one for epel?
You can use whichever URL you want. It just has to point to the top-level directory of the category. It has to point to the same place the HTTP URLs do.
There are a few problems with the URLs you configured.
For the category 'Fedora Linux' the HTTP and HTTPS URLs are missing '/linux'. The RSYNC URL is correct.
For the category 'Fedora EPEL', the RSYNC URL is also broken:
$ rsync rsync://espejito.fder.edu.uy/fedora/epel/epel rsync: link_stat "/epel/epel" (in fedora) failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1659) [Receiver=3.1.3]
There also seems to be a problem with your report_mirror setup, because once report_mirror (maybe as part of quick-fedora-mirror) runs, your mirror is also marked as not up to date. Just disabled running the reporting for now and the crawler will correctly scan your mirror.
Adrian
Thanks for your mail.
El 31/10/18 a las 5:33, Adrian Reber escribió:
There are a few problems with the URLs you configured. For the category 'Fedora Linux' the HTTP and HTTPS URLs are missing '/linux'. The RSYNC URL is correct.
Fixed
For the category 'Fedora EPEL', the RSYNC URL is also broken:
$ rsync rsync://espejito.fder.edu.uy/fedora/epel/epel rsync: link_stat "/epel/epel" (in fedora) failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1659) [Receiver=3.1.3]
Fixed
There also seems to be a problem with your report_mirror setup, because once report_mirror (maybe as part of quick-fedora-mirror) runs, your mirror is also marked as not up to date. Just disabled running the reporting for now and the crawler will correctly scan your mirror.
I tried to configure quick-fedora-mirror.sh to report. I configured the following variables:
CHECKIN_SITE=espejito CHECKIN_PASSWORD=**** CHECKIN_HOST=espejito.fder.edu.uy
this is the output of a run with verbosity 3
Mirror starting: Wed Oct 31 06:55:59 -03 2018 >> Log: Run start: cfg /root/bin/quick-fedora-mirror.conf, tmp /tmp/quick-mirror.QhWuzc0qWs ============================================================ >> Log: Remote file list download start Downloading file lists >> Calling /usr/bin/rsync --timeout=600 -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i %10l %n%L -v --files-from=filelist-transferlist --no-dirs --relative --compress rsync://mirrors.kernel.org/fedora-buffet/ . >> Log: calling /usr/bin/rsync --timeout=600 -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i %10l %n%L -v --files-from=filelist-transferlist --no-dirs --relative --compress rsync://mirrors.kernel.org/fedora-buffet/ . >> Log: rsync call completed succesfully with return 0 >> Log: File list download: 181B received, 119.20B/s >> Log: Remote file list download: end >> Log: Processing start >> Log: No change in file list for fedora-enchilada No change in file list checksum. Skipping fedora-enchilada. >> Log: No change in file list for fedora-epel No change in file list checksum. Skipping fedora-epel. >> Log: No changes to synchronize No changed files. ========================= Mirror finished: Wed Oct 31 06:56:03 -03 2018 (0) >> Log: Run end; exiting 0.
Do you see anything wrong?
regards
Adrian
On Wed, Oct 31, 2018 at 07:04:08AM -0300, ariel sabiguero yawelak wrote:
Thanks for your mail.
El 31/10/18 a las 5:33, Adrian Reber escribió:
There are a few problems with the URLs you configured. For the category 'Fedora Linux' the HTTP and HTTPS URLs are missing '/linux'. The RSYNC URL is correct.
Fixed
For the category 'Fedora EPEL', the RSYNC URL is also broken:
$ rsync rsync://espejito.fder.edu.uy/fedora/epel/epel rsync: link_stat "/epel/epel" (in fedora) failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1659) [Receiver=3.1.3]
Fixed
There also seems to be a problem with your report_mirror setup, because once report_mirror (maybe as part of quick-fedora-mirror) runs, your mirror is also marked as not up to date. Just disabled running the reporting for now and the crawler will correctly scan your mirror.
I tried to configure quick-fedora-mirror.sh to report. I configured the following variables:
CHECKIN_SITE=espejito CHECKIN_PASSWORD=**** CHECKIN_HOST=espejito.fder.edu.uy
this is the output of a run with verbosity 3
Mirror starting: Wed Oct 31 06:55:59 -03 2018 >> Log: Run start: cfg /root/bin/quick-fedora-mirror.conf, tmp /tmp/quick-mirror.QhWuzc0qWs ============================================================ >> Log: Remote file list download start Downloading file lists >> Calling /usr/bin/rsync --timeout=600 -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i %10l %n%L -v --files-from=filelist-transferlist --no-dirs --relative --compress rsync://mirrors.kernel.org/fedora-buffet/ . >> Log: calling /usr/bin/rsync --timeout=600 -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i %10l %n%L -v --files-from=filelist-transferlist --no-dirs --relative --compress rsync://mirrors.kernel.org/fedora-buffet/ . >> Log: rsync call completed succesfully with return 0 >> Log: File list download: 181B received, 119.20B/s >> Log: Remote file list download: end >> Log: Processing start >> Log: No change in file list for fedora-enchilada No change in file list checksum. Skipping fedora-enchilada. >> Log: No change in file list for fedora-epel No change in file list checksum. Skipping fedora-epel. >> Log: No changes to synchronize No changed files. ========================= Mirror finished: Wed Oct 31 06:56:03 -03 2018 (0) >> Log: Run end; exiting 0.
Do you see anything wrong?
No, but I don't know much about quick-fedora-mirror's reporting code.
Now that you URLs are correct and after I manually started a crawl of you mirror, everything looks good and you should see clients being re-directed to your mirror.
The output looks like you are running your sync script as root. That is not necessary.
Adrian
I forgot the "sudo" when I ran it last time.
In the crontab runs as user mirror. Just in case, I just chowned the whole branch to mirror.mirror.
we are serving content, properly right now. How can I check correctness of our reporting?
regards
ariel
El 31/10/18 a las 9:23, Adrian Reber escribió:
On Wed, Oct 31, 2018 at 07:04:08AM -0300, ariel sabiguero yawelak wrote:
Thanks for your mail.
El 31/10/18 a las 5:33, Adrian Reber escribió:
There are a few problems with the URLs you configured. For the category 'Fedora Linux' the HTTP and HTTPS URLs are missing '/linux'. The RSYNC URL is correct.
Fixed
For the category 'Fedora EPEL', the RSYNC URL is also broken:
$ rsync rsync://espejito.fder.edu.uy/fedora/epel/epel rsync: link_stat "/epel/epel" (in fedora) failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1659) [Receiver=3.1.3]
Fixed
There also seems to be a problem with your report_mirror setup, because once report_mirror (maybe as part of quick-fedora-mirror) runs, your mirror is also marked as not up to date. Just disabled running the reporting for now and the crawler will correctly scan your mirror.
I tried to configure quick-fedora-mirror.sh to report. I configured the following variables:
CHECKIN_SITE=espejito CHECKIN_PASSWORD=**** CHECKIN_HOST=espejito.fder.edu.uythis is the output of a run with verbosity 3
Mirror starting: Wed Oct 31 06:55:59 -03 2018 >> Log: Run start: cfg /root/bin/quick-fedora-mirror.conf, tmp /tmp/quick-mirror.QhWuzc0qWs ============================================================ >> Log: Remote file list download start Downloading file lists >> Calling /usr/bin/rsync --timeout=600 -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i %10l %n%L -v --files-from=filelist-transferlist --no-dirs --relative --compress rsync://mirrors.kernel.org/fedora-buffet/ . >> Log: calling /usr/bin/rsync --timeout=600 -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i %10l %n%L -v --files-from=filelist-transferlist --no-dirs --relative --compress rsync://mirrors.kernel.org/fedora-buffet/ . >> Log: rsync call completed succesfully with return 0 >> Log: File list download: 181B received, 119.20B/s >> Log: Remote file list download: end >> Log: Processing start >> Log: No change in file list for fedora-enchilada No change in file list checksum. Skipping fedora-enchilada. >> Log: No change in file list for fedora-epel No change in file list checksum. Skipping fedora-epel. >> Log: No changes to synchronize No changed files. ========================= Mirror finished: Wed Oct 31 06:56:03 -03 2018 (0) >> Log: Run end; exiting 0.Do you see anything wrong?
No, but I don't know much about quick-fedora-mirror's reporting code.
Now that you URLs are correct and after I manually started a crawl of you mirror, everything looks good and you should see clients being re-directed to your mirror.
The output looks like you are running your sync script as root. That is not necessary.
Adrian
On Wed, Oct 31, 2018 at 09:42:24AM -0300, ariel sabiguero yawelak wrote:
I forgot the "sudo" when I ran it last time.
In the crontab runs as user mirror. Just in case, I just chowned the whole branch to mirror.mirror.
we are serving content, properly right now. How can I check correctness of our reporting?
Reporting is optional. If you run the report_mirror tool and paste the output here I can probably tell you if it is correct. Unfortunately I don't know how quick-fedora-mirror's reporting works.
Adrian
This is the output:
espejito:~/bin # sudo -u mirror ./report_mirror Category Fedora Linux directories updated: 0 added: 0 deleted 6080 Category Fedora EPEL directories updated: 1000 added: 0 deleted 0 checked in successful
It seems that it removed a lot of directories...
This is the report_mirror.conf
espejito:~/bin # cat report_mirror.conf | grep -v ^# | grep -v ^$ [global] enabled=1 server=https://admin.fedoraproject.org/mirrormanager/xmlrpc [site] enabled=1 name=espejito password=**** [host] enabled=1 name=espejito.fder.edu.uy [stats] enabled=0 apache=/var/log/httpd/access_log vsftpd=/var/log/vsftpd.log rsyncd=/var/log/rsyncd.log [Fedora Linux] enabled=1 path=/zp0/mirrors/fedora/fedora/linux [Fedora EPEL] enabled=1 path=/zp0/mirrors/fedora/epel [Fedora Secondary Arches] enabled=0 path=/var/www/html/pub/fedora-secondary [Fedora Other] enabled=0 path=/var/www/html/pub/alt [Fedora Archive] enabled=0 path=/var/www/html/pub/fedora-archive
regards
ariel
El 31/10/18 a las 9:50, Adrian Reber escribió:
On Wed, Oct 31, 2018 at 09:42:24AM -0300, ariel sabiguero yawelak wrote:
I forgot the "sudo" when I ran it last time.
In the crontab runs as user mirror. Just in case, I just chowned the whole branch to mirror.mirror.
we are serving content, properly right now. How can I check correctness of our reporting?
Reporting is optional. If you run the report_mirror tool and paste the output here I can probably tell you if it is correct. Unfortunately I don't know how quick-fedora-mirror's reporting works.
Adrian
On Wed, Oct 31, 2018 at 10:19:48AM -0300, ariel sabiguero yawelak wrote:
This is the output:
espejito:~/bin # sudo -u mirror ./report_mirror Category Fedora Linux directories updated: 0 added: 0 deleted 6080 Category Fedora EPEL directories updated: 1000 added: 0 deleted 0 checked in successful
It seems that it removed a lot of directories...
Yes, you removed everything from the database. Your "Fedora Linux" category is now completely empty.
This is the report_mirror.conf
espejito:~/bin # cat report_mirror.conf | grep -v ^# | grep -v ^$ [global] enabled=1 server=https://admin.fedoraproject.org/mirrormanager/xmlrpc [site] enabled=1 name=espejito password=**** [host] enabled=1 name=espejito.fder.edu.uy [stats] enabled=0 apache=/var/log/httpd/access_log vsftpd=/var/log/vsftpd.log rsyncd=/var/log/rsyncd.log [Fedora Linux] enabled=1 path=/zp0/mirrors/fedora/fedora/linux [Fedora EPEL] enabled=1 path=/zp0/mirrors/fedora/epel [Fedora Secondary Arches] enabled=0 path=/var/www/html/pub/fedora-secondary [Fedora Other] enabled=0 path=/var/www/html/pub/alt [Fedora Archive] enabled=0 path=/var/www/html/pub/fedora-archive
The config looks correct, but it is hard to tell without knowing what is on your system. Just disable reporting and don't worry about it.
Adrian
I just removed credentials from quick-fedora-mirror script and commented out report mirror.
I would like to be able to correct the issue. Can I provide you additional information?
El 31/10/18 a las 10:32, Adrian Reber escribió:
On Wed, Oct 31, 2018 at 10:19:48AM -0300, ariel sabiguero yawelak wrote:
This is the output:
espejito:~/bin # sudo -u mirror ./report_mirror Category Fedora Linux directories updated: 0 added: 0 deleted 6080 Category Fedora EPEL directories updated: 1000 added: 0 deleted 0 checked in successfulIt seems that it removed a lot of directories...
Yes, you removed everything from the database. Your "Fedora Linux" category is now completely empty.
This is the report_mirror.conf
espejito:~/bin # cat report_mirror.conf | grep -v ^# | grep -v ^$ [global] enabled=1 server=https://admin.fedoraproject.org/mirrormanager/xmlrpc [site] enabled=1 name=espejito password=**** [host] enabled=1 name=espejito.fder.edu.uy [stats] enabled=0 apache=/var/log/httpd/access_log vsftpd=/var/log/vsftpd.log rsyncd=/var/log/rsyncd.log [Fedora Linux] enabled=1 path=/zp0/mirrors/fedora/fedora/linux [Fedora EPEL] enabled=1 path=/zp0/mirrors/fedora/epel [Fedora Secondary Arches] enabled=0 path=/var/www/html/pub/fedora-secondary [Fedora Other] enabled=0 path=/var/www/html/pub/alt [Fedora Archive] enabled=0 path=/var/www/html/pub/fedora-archiveThe config looks correct, but it is hard to tell without knowing what is on your system. Just disable reporting and don't worry about it.
Adrian
I am watching our log and it seems that we only serve epel content and no fedora at all.
What can I try to do? there seems to be something wrong on the other branch...
regards
ariel
El 31/10/18 a las 10:52, ariel sabiguero yawelak escribió:
I just removed credentials from quick-fedora-mirror script and commented out report mirror.
I would like to be able to correct the issue. Can I provide you additional information?
El 31/10/18 a las 10:32, Adrian Reber escribió:
On Wed, Oct 31, 2018 at 10:19:48AM -0300, ariel sabiguero yawelak wrote:
This is the output:
espejito:~/bin # sudo -u mirror ./report_mirror Category Fedora Linux directories updated: 0 added: 0 deleted 6080 Category Fedora EPEL directories updated: 1000 added: 0 deleted 0 checked in successful
It seems that it removed a lot of directories...
Yes, you removed everything from the database. Your "Fedora Linux" category is now completely empty.
This is the report_mirror.conf
espejito:~/bin # cat report_mirror.conf | grep -v ^# | grep -v ^$ [global] enabled=1 server=https://admin.fedoraproject.org/mirrormanager/xmlrpc [site] enabled=1 name=espejito password=**** [host] enabled=1 name=espejito.fder.edu.uy [stats] enabled=0 apache=/var/log/httpd/access_log vsftpd=/var/log/vsftpd.log rsyncd=/var/log/rsyncd.log [Fedora Linux] enabled=1 path=/zp0/mirrors/fedora/fedora/linux [Fedora EPEL] enabled=1 path=/zp0/mirrors/fedora/epel [Fedora Secondary Arches] enabled=0 path=/var/www/html/pub/fedora-secondary [Fedora Other] enabled=0 path=/var/www/html/pub/alt [Fedora Archive] enabled=0 path=/var/www/html/pub/fedora-archive
The config looks correct, but it is hard to tell without knowing what is on your system. Just disable reporting and don't worry about it.
Adrian
Mirror-admin mailing list -- mirror-admin@lists.fedoraproject.org To unsubscribe send an email to mirror-admin-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/mirror-admin@lists.fedoraproje...
mirror-admin@lists.fedoraproject.org
-
Adrian Reber -
ariel sabiguero yawelak