Hi,
I'm one of the admins of ftp.halifax.rwth-aachen.de, which offers fedora and fedora-epel among other distributions and projects. I've recently noticed that many Fedora users (including EPEL, CentOS, and BlackArch) frequently re-download "repodata" files that haven't been updated since the previous request.
As some of those files are rather large ("filelists") and others are hit extremely often, I added fail2ban rules to deny users access to our service for some time.
Please adjust your software so that file mirrors like ours are not (ab)used like this. Files that haven't been changed usually shouldn't be downloaded.
Thanks Carsten
On Tue, Apr 02, 2024 at 12:35:53PM +0200, Carsten Otto wrote:
Hi,
Hello.
I'm one of the admins of ftp.halifax.rwth-aachen.de, which offers fedora and fedora-epel among other distributions and projects. I've recently noticed that many Fedora users (including EPEL, CentOS, and BlackArch) frequently re-download "repodata" files that haven't been updated since the previous request.
How are you determining this? From IP address? Those could be a large number of different machines behind a nat proxy no?
As some of those files are rather large ("filelists") and others are hit extremely often, I added fail2ban rules to deny users access to our service for some time.
Recent versions of fedora do not even download the filelists unless a user requests/looks up something that needs it. For normal install or updates, it shouldn't be pulled. Can you tell what OS/versions this is happening with?
In general clients should pull the repomd.xml file and check if it's changed, if not, use their cached versions.
Please adjust your software so that file mirrors like ours are not (ab)used like this. Files that haven't been changed usually shouldn't be downloaded.
This should normally already be the case, so I wonder what is happening here, could be a bug in some version/os or otherwise something unexpected.
kevin
Hi Kevin,
On Tue, Apr 02, 2024 at 05:08:33PM -0700, Kevin Fenzi wrote:
How are you determining this? From IP address?
Yes.
Those could be a large number of different machines behind a nat proxy no?
Possibly.
Recent versions of fedora do not even download the filelists unless a user requests/looks up something that needs it. For normal install or updates, it shouldn't be pulled. Can you tell what OS/versions this is happening with?
Great! No, I just see basic file download information. Blocking those who do it much too often shouldn't be a concern then.
In general clients should pull the repomd.xml file and check if it's changed, if not, use their cached versions.
That's not what I see for quite a lot of peers.
This should normally already be the case, so I wonder what is happening here, could be a bug in some version/os or otherwise something unexpected.
That's my guess too.
Bye Carsten
On Tue, Apr 02, 2024 at 05:08:33PM -0700, Kevin Fenzi wrote:
In general clients should pull the repomd.xml file and check if it's changed, if not, use their cached versions.
Does this also apply to this file? /fedora-epel/7/x86_64/repodata/f25bccaa54b08e731162aec431dd82887a5ed1abce8eb6f892ad41454f07e1f3-primary.sqlite.bz2
I have five hosts which each downloaded 4 GByte of this file in the span of 24 hours. The delay between two downloads varies, from a few seconds to a few minutes.
On Wed, 3 Apr 2024 at 04:33, Carsten Otto otto@informatik.rwth-aachen.de wrote:
On Tue, Apr 02, 2024 at 05:08:33PM -0700, Kevin Fenzi wrote:
In general clients should pull the repomd.xml file and check if it's changed, if not, use their cached versions.
Does this also apply to this file?
/fedora-epel/7/x86_64/repodata/f25bccaa54b08e731162aec431dd82887a5ed1abce8eb6f892ad41454f07e1f3-primary.sqlite.bz2
I have five hosts which each downloaded 4 GByte of this file in the span of 24 hours. The delay between two downloads varies, from a few seconds to a few minutes.
Can you give the full httpd string (remove the ip address). This might help figure out if it is a specific version of a tool which is the problem or if it is a generic issue.
-- Dr. Carsten Otto http://verify.rwth-aachen.de/otto/ _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org https://lists.centos.org/mailman/listinfo/centos-mirror
On Wed, Apr 03, 2024 at 08:38:17AM -0400, Stephen Smoogen wrote:
Can you give the full httpd string (remove the ip address). This might help figure out if it is a specific version of a tool which is the problem or if it is a generic issue.
I don't log user agents, sorry. I could temporarily enable that if helps?
On Wed, 3 Apr 2024 at 08:42, Carsten Otto otto@informatik.rwth-aachen.de wrote:
On Wed, Apr 03, 2024 at 08:38:17AM -0400, Stephen Smoogen wrote:
Can you give the full httpd string (remove the ip address). This might
help
figure out if it is a specific version of a tool which is the problem or
if
it is a generic issue.
I don't log user agents, sorry. I could temporarily enable that if helps?
If you for debugging purposes and then turn it off again it would be helpful. There are multiple tools which can download this data from a 'client' point of view and some of them are going to be 'buggy'.
-- Dr. Carsten Otto http://verify.rwth-aachen.de/otto/
On Wed, Apr 03, 2024 at 08:45:57AM -0400, Stephen Smoogen wrote:
If you for debugging purposes and then turn it off again it would be helpful. There are multiple tools which can download this data from a 'client' point of view and some of them are going to be 'buggy'.
Downloads a few files about once a minute:
libdnf (Rocky Linux 8.8; generic; Linux.x86_64)
/fedora-epel/8/Everything/x86_64/repodata/repomd.xml /fedora-epel/8/Everything/x86_64/repodata/0e32b3c58c14790acdb973f20a4b2207e24720222513b0bc63297a649a9884a8-comps-Everything.x86_64.xml /fedora-epel/8/Everything/x86_64/repodata/cf47b564c8fa629069ef0f0836dda95fff05e077068eb3e9ed554c343581be84-prestodelta.xml.xz /fedora-epel/8/Everything/x86_64/repodata/6f0dd08e0df456c5c28d4f776aa71b4bb6d0e4bce4e4bf3423260efdd4b7af2b-updateinfo.xml.bz2 /fedora-epel/8/Everything/x86_64/repodata/1f3bb463d1f38d18f6af974e8353b7e8ad7f368686a462c94d664541f00c7c39-primary.xml.gz /fedora-epel/8/Everything/x86_64/repodata/20761194528ef5ba68ccab20ed7ab3f9303c81ebc562f9dca772a7f20db59dd8-filelists.xml.gz
Similar issue, different user:
libdnf (Rocky Linux 9.3; generic; Linux.x86_64)
/fedora-epel/9/Everything/x86_64/repodata/repomd.xml /fedora-epel/9/Everything/x86_64/repodata/8768391f4f5cb0daebfd4f6103704b325933733c458df8f261b1e47601f81afc-comps-Everything.x86_64.xml.gz /fedora-epel/9/Everything/x86_64/repodata/881a09d92cdb4078eed6995d8e2c48304c4a36011b6fdc695bc19705f86ab2fe-updateinfo.xml.bz2 /fedora-epel/9/Everything/x86_64/repodata/a38ae61a2abfb200cdc4a796f5478c8bcbba4b6a222ab46e7893d9879760734a-primary.xml.gz /fedora-epel/9/Everything/x86_64/repodata/b2cd153314a9aed419fa6adf682c85503be7b6045772bf6cfffef2a4f6fd19fb-filelists.xml.gz
This user downloads other files, also about once a minute (via HTTP 2.0):
urlgrabber/3.10 yum/3.4.3
/fedora-epel/7/x86_64/repodata/repomd.xml /fedora-epel/7/x86_64/repodata/b067b70cf45ef66ab62dc5661b802249f45756ef620528ccf9843d04ca6da83b-updateinfo.xml.bz2 /fedora-epel/7/x86_64/repodata/11ce2dc3bd977b1cbb9acccf51629f6d07ee9c191fecd960ee82df1d307ef3cb-primary.sqlite.bz2
I'll provide more examples as soon as they pour in.
Every few minutes:
libdnf (Red Hat Enterprise Linux 8.6; generic; Linux.x86_64)
/fedora-epel/8/Everything/x86_64/repodata/repomd.xml /fedora-epel/8/Everything/x86_64/repodata/0e32b3c58c14790acdb973f20a4b2207e24720222513b0bc63297a649a9884a8-comps-Everything.x86_64.xml /fedora-epel/8/Everything/x86_64/repodata/cf47b564c8fa629069ef0f0836dda95fff05e077068eb3e9ed554c343581be84-prestodelta.xml.xz /fedora-epel/8/Everything/x86_64/repodata/6f0dd08e0df456c5c28d4f776aa71b4bb6d0e4bce4e4bf3423260efdd4b7af2b-updateinfo.xml.bz2 /fedora-epel/8/Everything/x86_64/repodata/1f3bb463d1f38d18f6af974e8353b7e8ad7f368686a462c94d664541f00c7c39-primary.xml.gz /fedora-epel/8/Everything/x86_64/repodata/20761194528ef5ba68ccab20ed7ab3f9303c81ebc562f9dca772a7f20db59dd8-filelists.xml.gz
More examples (similar patterns to those already shown):
libdnf (AlmaLinux 8.8; generic; Linux.x86_64) libdnf (Rocky Linux 8.8; generic; Linux.x86_64) libdnf (Rocky Linux 8.9; generic; Linux.x86_64) libdnf (Rocky Linux 9.2; generic; Linux.x86_64) urlgrabber/3.10 yum/3.4.3 (HTTP 2.0)
On Tue, Apr 2, 2024 at 7:08 PM Kevin Fenzi kevin@scrye.com wrote:
On Tue, Apr 02, 2024 at 12:35:53PM +0200, Carsten Otto wrote:
Hi,
Hello.
I'm one of the admins of ftp.halifax.rwth-aachen.de, which offers fedora and fedora-epel among other distributions and projects. I've recently noticed that many Fedora users (including EPEL, CentOS, and BlackArch) frequently re-download "repodata" files that haven't been updated since the previous request.
How are you determining this? From IP address? Those could be a large number of different machines behind a nat proxy no?
As some of those files are rather large ("filelists") and others are hit extremely often, I added fail2ban rules to deny users access to our service for some time.
Recent versions of fedora do not even download the filelists unless a user requests/looks up something that needs it. For normal install or updates, it shouldn't be pulled. Can you tell what OS/versions this is happening with?
In general clients should pull the repomd.xml file and check if it's changed, if not, use their cached versions.
An exception to this would be people building containers. Theytypically remove the metadata at the end of each build. So do not have it cached. However, the described behaviour would seem to be more widespread than some people building containers, which may indicate some other reason that a group of people are repeatedly grabbing the same data. It would be interesting to know if they are only getting repodata or also downloading rpms and other things.
Dennis
Please adjust your software so that file mirrors like ours are not (ab)used like this. Files that haven't been changed usually shouldn't be downloaded.
This should normally already be the case, so I wonder what is happening here, could be a bug in some version/os or otherwise something unexpected.
kevin
Mirror-admin mailing list -- mirror-admin@lists.fedoraproject.org To unsubscribe send an email to mirror-admin-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/mirror-admin@lists.fedoraproje... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Wed, Apr 03, 2024 at 12:54:10PM -0500, Dennis Gilmore wrote:
An exception to this would be people building containers. Theytypically remove the metadata at the end of each build. So do not have it cached. However, the described behaviour would seem to be more widespread than some people building containers, which may indicate some other reason that a group of people are repeatedly grabbing the same data. It would be interesting to know if they are only getting repodata or also downloading rpms and other things.
Interesting thought! I just grepped for a few of the "abusive" hosts. These hosts are downloading packages, including some "basic" ones like perl, curl, python setuptools.
I'll relax the fail2ban rules a bit and see how things evolve over time. Maybe everyone needs to rebuild everything once now, which obviously causes a bit of traffic.
On Wed, 3 Apr 2024 at 14:08, Carsten Otto otto@informatik.rwth-aachen.de wrote:
On Wed, Apr 03, 2024 at 12:54:10PM -0500, Dennis Gilmore wrote:
An exception to this would be people building containers. Theytypically remove the metadata at the end of each build. So do not have it cached. However, the described behaviour would seem to be more widespread than some people building containers, which may indicate some other reason that a group of people are repeatedly grabbing the same data. It would be interesting to know if they are only getting repodata or also downloading rpms and other things.
Interesting thought! I just grepped for a few of the "abusive" hosts. These hosts are downloading packages, including some "basic" ones like perl, curl, python setuptools.
I'll relax the fail2ban rules a bit and see how things evolve over time. Maybe everyone needs to rebuild everything once now, which obviously causes a bit of traffic.
My expectation is that traffic is going to be higher in the next 6 months as various systems are getting moved from CentOS 7 to some newer release. This upgrade process will require a system to be first updated to the latest '7' and many have had updates turned off for years.. then there will be a download and update to EL8 or EL9 depending on what they are doing which will also create a spike in traffic.
-- Dr. Carsten Otto http://verify.rwth-aachen.de/otto/ -- _______________________________________________ Mirror-admin mailing list -- mirror-admin@lists.fedoraproject.org To unsubscribe send an email to mirror-admin-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/mirror-admin@lists.fedoraproje... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Wed, Apr 03, 2024 at 02:14:07PM -0400, Stephen John Smoogen wrote:
My expectation is that traffic is going to be higher in the next 6 months as various systems are getting moved from CentOS 7 to some newer release. This upgrade process will require a system to be first updated to the latest '7' and many have had updates turned off for years.. then there will be a download and update to EL8 or EL9 depending on what they are doing which will also create a spike in traffic.
Even better, that sounds like useful traffic and I have around 15 GBit/sec to spare for that :)
I'll just chime in with what we have been seeing.
We run mirror servers in multiple countries (US, UK, SG, AU), and ALL of the EPEL mirrors have been getting hit with DDoS Flood Attacks.
One of the Micro mirrors we had to shut down completely here in the US.
On 2024-04-02 10:35, Carsten Otto wrote:
Hi,
I'm one of the admins of ftp.halifax.rwth-aachen.de, which offers fedora and fedora-epel among other distributions and projects. I've recently noticed that many Fedora users (including EPEL, CentOS, and BlackArch) frequently re-download "repodata" files that haven't been updated since the previous request.
As some of those files are rather large ("filelists") and others are hit extremely often, I added fail2ban rules to deny users access to our service for some time.
Please adjust your software so that file mirrors like ours are not (ab)used like this. Files that haven't been changed usually shouldn't be downloaded.
Thanks Carsten
-- _______________________________________________ Mirror-admin mailing list -- mirror-admin@lists.fedoraproject.org To unsubscribe send an email to mirror-admin-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/mirror-admin@lists.fedoraproje... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Any chance some of the volume changes are due to patching/migrations related to the recent SSH vulnerability?
-Paulson
On Wed, Apr 3, 2024 at 12:01 PM Starburst Services SysOp Team via Mirror-admin mirror-admin@lists.fedoraproject.org wrote:
I'll just chime in with what we have been seeing.
We run mirror servers in multiple countries (US, UK, SG, AU), and ALL of the EPEL mirrors have been getting hit with DDoS Flood Attacks.
One of the Micro mirrors we had to shut down completely here in the US.
On 2024-04-02 10:35, Carsten Otto wrote:
Hi,
I'm one of the admins of ftp.halifax.rwth-aachen.de, which offers fedora and fedora-epel among other distributions and projects. I've recently noticed that many Fedora users (including EPEL, CentOS, and BlackArch) frequently re-download "repodata" files that haven't been updated since the previous request.
As some of those files are rather large ("filelists") and others are hit extremely often, I added fail2ban rules to deny users access to our service for some time.
Please adjust your software so that file mirrors like ours are not (ab)used like this. Files that haven't been changed usually shouldn't be downloaded.
Thanks Carsten
-- _______________________________________________ Mirror-admin mailing list -- mirror-admin@lists.fedoraproject.org To unsubscribe send an email to
mirror-admin-leave@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/mirror-admin@lists.fedoraproje...
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
Mirror-admin mailing list -- mirror-admin@lists.fedoraproject.org To unsubscribe send an email to mirror-admin-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/mirror-admin@lists.fedoraproje... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Wed, Apr 03, 2024 at 12:16:56PM -0400, Paulson McIntyre wrote:
Any chance some of the volume changes are due to patching/migrations related to the recent SSH vulnerability?
I tried, but I wasn't able to find any outstanding file or host. It looks like it's more traffic all over the board, which I guess happens if lots of users need to update lots of packages.
I do have lots of old logs, do you know of a few specific files that I could check?
Interesting question....The repo metadata obviously, sshd, and xz/lzma libs. Perhaps anything that depends on those three libs though.
-Paulson
On Wed, Apr 3, 2024 at 12:27 PM Carsten Otto otto@informatik.rwth-aachen.de wrote:
On Wed, Apr 03, 2024 at 12:16:56PM -0400, Paulson McIntyre wrote:
Any chance some of the volume changes are due to patching/migrations related to the recent SSH vulnerability?
I tried, but I wasn't able to find any outstanding file or host. It looks like it's more traffic all over the board, which I guess happens if lots of users need to update lots of packages.
I do have lots of old logs, do you know of a few specific files that I could check? -- Dr. Carsten Otto http://verify.rwth-aachen.de/otto/
On Wed, Apr 03, 2024 at 12:44:41PM -0400, Paulson McIntyre wrote:
Interesting question....The repo metadata obviously, sshd, and xz/lzma libs. Perhaps anything that depends on those three libs though.
I don't see any noteworthy change for fedora-epel downloads of files including "ssh" (in March).
No changes in sshd
That's actually a bit scary given the seriousness of the ssh vuln.
-Paulson
On Wed, Apr 3, 2024 at 1:10 PM Carsten Otto otto@informatik.rwth-aachen.de wrote:
On Wed, Apr 03, 2024 at 12:44:41PM -0400, Paulson McIntyre wrote:
Interesting question....The repo metadata obviously, sshd, and xz/lzma libs. Perhaps anything that depends on those three libs though.
I don't see any noteworthy change for fedora-epel downloads of files including "ssh" (in March). -- Dr. Carsten Otto http://verify.rwth-aachen.de/otto/
On Wed, Apr 03, 2024 at 01:20:31PM -0400, Paulson McIntyre wrote:
No changes in sshd
That's actually a bit scary given the seriousness of the ssh vuln.
RHEL/CentOS/Rocky/Alma/etc, never had the vulnerable version. It was never even built for them. EPEL doesn't provide xz (thats in the base EL repos).
If people are checking for updates, that could explain some increased traffic, but I wouldn't think it would explain any dramatic traffic increase.
A number of other mirror admins don't seem to be seeing anything different, so it sounds to me like it might be region specific perhaps?
Are all these requests ipv4? Are they coming from the same or different providers?
I can try and look for patterns in the master mirror logs...
kevin
On Wed, Apr 03, 2024 at 10:37:20AM -0700, Kevin Fenzi wrote:
RHEL/CentOS/Rocky/Alma/etc, never had the vulnerable version. It was never even built for them. EPEL doesn't provide xz (thats in the base EL repos).
Ah, that explains the lack of actual downloads then :)
If people are checking for updates, that could explain some increased traffic, but I wouldn't think it would explain any dramatic traffic increase.
It's about 10x, not dramatic, but still quite a lot - especially because it's just metadata from the looks of it.
A number of other mirror admins don't seem to be seeing anything different, so it sounds to me like it might be region specific perhaps?
Are all these requests ipv4? Are they coming from the same or different providers?
As far as I can tell, it's all over the place. A few hosts are outliers (maybe due to NAT or impatience), but overall, it's just more of everything.
So, to summarize, the idea is that due to the xz/ssh issue, lots of users are downloading metadata, especially the "filelists", causing an ongoing surge in traffic.
If that's true, things should calm down anytime now.
OK, I dug in deeper, a few more insights.
end of February / early March: - 30k-40k hits for /fedora-epel/8/Everything/x86_64/repodata/repomd.xml, comps-Everything.x86_64.xml, -filelists.xml.gz, ... - 10k hits or so for the same in /7/
end of March / early April: - around 30k hits for /8/ and /9/ files - 400k-700k (!) for /7/
Top hitters: /fedora-epel/7/x86_64/repodata/37f85d91d087d574c4551a96731a6e8f2d6b2a458d364901fdeb7858e6fc7950-updateinfo.xml.bz2 /fedora-epel/7/x86_64/repodata/37f85d91d087d574c4551a96731a6e8f2d6b2a458d364901fdeb7858e6fc7950-updateinfo.xml.bz2 /fedora-epel/7/x86_64/repodata/6111d049c4d53fc00176b3da63856b580c6a34f1b7a2062ccf821988d095ec44-updateinfo.xml.bz2 /fedora-epel/7/x86_64/repodata/6fa65768495a2a46eff12c510acc23d4df39a85df14958ac5bfea0f758d9a43a-primary.sqlite.bz2 /fedora-epel/7/x86_64/repodata/e52552f46d9f450b7b5966d4cdafdc58769eda686fc2d9a4c44beb441b122453-primary.sqlite.bz2 /fedora-epel/7/x86_64/repodata/e52552f46d9f450b7b5966d4cdafdc58769eda686fc2d9a4c44beb441b122453-primary.sqlite.bz2 /fedora-epel/7/x86_64/repodata/repomd.xml /fedora-epel/7/x86_64/repodata/repomd.xml /fedora-epel/7/x86_64/repodata/repomd.xml
It looks like the number of requests for the /7/ files picked up during March, but a sudden increase happened around March 28th.
It looks like traffic jumped up due to downloads of "-filelists.xml" files at that time, but I also see lots of traffic for "-primary.sqlite" and "-updateinfo".
That's about the time when the libxz mess become public.
When did you start shipping security updates to SSH/liblzma/...?
Hi again,
yesterday something happened, the fedora-epel load on our mirror suddenly dropped to almost zero - see attached images.
I don't know what's going on.
Bye Carsten
On Fri, Apr 12, 2024 at 08:29:59AM +0200, Carsten Otto wrote:
yesterday something happened, the fedora-epel load on our mirror suddenly dropped to almost zero - see attached images.
Your mirror was not up to date and was dropped from the list of up to date mirrors, the crawler already detected that your mirror is up to date again and in about 30 minutes it should see traffic again.
Adrian
On Fri, Apr 12, 2024 at 08:51:28AM +0200, Adrian Reber wrote:
Your mirror was not up to date and was dropped from the list of up to date mirrors, the crawler already detected that your mirror is up to date again and in about 30 minutes it should see traffic again.
Oh, interesting. Should I update more frequently than every two hours? Do you offer SSH push triggers / rsync pushes?
Thanks Carsten
On Fri, Apr 12, 2024 at 09:08:37AM +0200, Carsten Otto wrote:
On Fri, Apr 12, 2024 at 08:51:28AM +0200, Adrian Reber wrote:
Your mirror was not up to date and was dropped from the list of up to date mirrors, the crawler already detected that your mirror is up to date again and in about 30 minutes it should see traffic again.
Oh, interesting. Should I update more frequently than every two hours?
It depends. If you are using quick-fedora-mirror you can do it every 10 minutes. Using full rsync would result in higher I/O load. MirrorManager scans EPEL mirrors every 6 hours, so there is always the possibility to be removed from the list of active mirrors.
With the help of quick-fedora-mirror I am running my mirror scripts every hour, but it takes between 2 and 5 seconds to decide if the primary mirror has changed. Just using rsync would probably require more time. So I think every two hours sounds right.
Do you offer SSH push triggers / rsync pushes?
No.
Adrian
Hi,
I want to raise a new issue here, is the origin server has TCP settings such as BBR and buffer settings. If not, is it possible to set them up?
It is quite slow to sync the contents when there is high latency. Our solution ends up - open an VPS near the origin server - configure with those setting - have a TCP proxy using socat and which makes us the only server in north east asia still updated during that time.
We also observe traffic increase and those traffic are coming from other countries. [image: image.png] [image: image.png]
Thanks ᐧ
On Fri, Apr 12, 2024 at 5:55 PM Adrian Reber via Mirror-admin < mirror-admin@lists.fedoraproject.org> wrote:
On Fri, Apr 12, 2024 at 09:08:37AM +0200, Carsten Otto wrote:
On Fri, Apr 12, 2024 at 08:51:28AM +0200, Adrian Reber wrote:
Your mirror was not up to date and was dropped from the list of up to date mirrors, the crawler already detected that your mirror is up to date again and in about 30 minutes it should see traffic again.
Oh, interesting. Should I update more frequently than every two hours?
It depends. If you are using quick-fedora-mirror you can do it every 10 minutes. Using full rsync would result in higher I/O load. MirrorManager scans EPEL mirrors every 6 hours, so there is always the possibility to be removed from the list of active mirrors.
With the help of quick-fedora-mirror I am running my mirror scripts every hour, but it takes between 2 and 5 seconds to decide if the primary mirror has changed. Just using rsync would probably require more time. So I think every two hours sounds right.
Do you offer SSH push triggers / rsync pushes?
No.
Adrian-- _______________________________________________ Mirror-admin mailing list -- mirror-admin@lists.fedoraproject.org To unsubscribe send an email to mirror-admin-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/mirror-admin@lists.fedoraproje... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Fri, Apr 12, 2024 at 09:05:24PM +0800, Jasper Yu wrote:
We also observe traffic increase and those traffic are coming from other countries.
Unfortunately I don't know where your mirror is located. I also do not know the timezone on your diagrams and what is causing the traffic exactly, but if you look at the timestamps of EPEL you can see that today around 3am (UTC) a new compose was created and pushed to the mirrors, that will result in more clients downloading the changed metadata and potentially the corresponding updates from mirrors. If there are not many mirror in your region this can result in a lot of traffic.
Adrian
mirror-admin@lists.fedoraproject.org
-
Adrian Reber -
Carsten Otto -
Dennis Gilmore -
Jasper Yu -
Kevin Fenzi -
Paulson McIntyre -
Starburst Services SysOp Team -
Stephen John Smoogen -
Stephen Smoogen