Abusive repodata downloads

[Carsten Otto]

Hi,

I'm one of the admins of ftp.halifax.rwth-aachen.de, which offers fedora
and fedora-epel among other distributions and projects. I've recently
noticed that many Fedora users (including EPEL, CentOS, and BlackArch)
frequently re-download "repodata" files that haven't been updated since
the previous request.

As some of those files are rather large ("filelists") and others are hit
extremely often, I added fail2ban rules to deny users access to our
service for some time.

Please adjust your software so that file mirrors like ours are not
(ab)used like this. Files that haven't been changed usually shouldn't be
downloaded.

Thanks
Carsten
-- 
Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

[Kevin Fenzi]

On Tue, Apr 02, 2024 at 12:35:53PM +0200, Carsten Otto wrote:
>  Hi, 
Hello.

>  I'm one of the admins of ftp.halifax.rwth-aachen.de, which offers
> fedora
>  and fedora-epel among other distributions and projects. I've recently
>  noticed that many Fedora users (including EPEL, CentOS, and BlackArch)
>  frequently re-download "repodata" files that haven't been updated since
>  the previous request. 
How are you determining this? From IP address?
Those could be a large number of different machines behind a nat proxy
no?

>  As some of those files are rather large ("filelists") and
> others are hit
>  extremely often, I added fail2ban rules to deny users access to our
>  service for some time. 
Recent versions of fedora do not even download the filelists unless a
user requests/looks up something that needs it. For normal install or
updates, it shouldn't be pulled. Can you tell what OS/versions this is
happening with?

In general clients should pull the repomd.xml file and check if it's
changed, if not, use their cached versions.

>  Please adjust your software so that file mirrors like ours are not
>  (ab)used like this. Files that haven't been changed usually shouldn't be
>  downloaded. 
This should normally already be the case, so I wonder what is happening
here, could be a bug in some version/os or otherwise something
unexpected.

kevin

[Carsten Otto]

Hi Kevin,

On Tue, Apr 02, 2024 at 05:08:33PM -0700, Kevin Fenzi wrote:
>  How are you determining this? From IP address? 
Yes.

>  Those could be a large number of different machines behind a nat
> proxy
>  no? 
Possibly.

>  Recent versions of fedora do not even download the filelists unless
> a
>  user requests/looks up something that needs it. For normal install or
>  updates, it shouldn't be pulled. Can you tell what OS/versions this is
>  happening with? 
Great! No, I just see basic file download information. Blocking those
who do it much too often shouldn't be a concern then.

>  In general clients should pull the repomd.xml file and check if
> it's
>  changed, if not, use their cached versions. 
That's not what I see for quite a lot of peers.

>  This should normally already be the case, so I wonder what is
> happening
>  here, could be a bug in some version/os or otherwise something
>  unexpected. 
That's my guess too.

Bye
Carsten
-- 
Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

[Carsten Otto]

On Tue, Apr 02, 2024 at 05:08:33PM -0700, Kevin Fenzi wrote:
>  In general clients should pull the repomd.xml file and check if
> it's
>  changed, if not, use their cached versions. 
Does this also apply to this file?
/fedora-epel/7/x86_64/repodata/f25bccaa54b08e731162aec431dd82887a5ed1abce8eb6f892ad41454f07e1f3-primary.sqlite.bz2

I have five hosts which each downloaded 4 GByte of this file in the span
of 24 hours. The delay between two downloads varies, from a few seconds
to a few minutes.
-- 
Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

[CentOS-mirror] Abusive repodata downloads

[Stephen Smoogen]

On Wed, 3 Apr 2024 at 04:33, Carsten Otto <otto(a)informatik.rwth-aachen.de&gt;
wrote:

>  On Tue, Apr 02, 2024 at 05:08:33PM -0700, Kevin Fenzi wrote:
>  > In general clients should pull the repomd.xml file and check if it's
>  > changed, if not, use their cached versions.
>
>  Does this also apply to this file?
>
>
> /fedora-epel/7/x86_64/repodata/f25bccaa54b08e731162aec431dd82887a5ed1abce8eb6f892ad41454f07e1f3-primary.sqlite.bz2
>
>  I have five hosts which each downloaded 4 GByte of this file in the span
>  of 24 hours. The delay between two downloads varies, from a few seconds
>  to a few minutes.
>  
Can you give the full httpd string (remove the ip address). This might help
figure out if it is a specific version of a tool which is the problem or if
it is a generic issue.



>  --
>  Dr. Carsten Otto
>  http://verify.rwth-aachen.de/otto/
>  _______________________________________________
>  CentOS-mirror mailing list
>  CentOS-mirror(a)centos.org
>  https://lists.centos.org/mailman/listinfo/centos-mirror
>  

-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren

[CentOS-mirror] Abusive repodata downloads

[Carsten Otto]

On Wed, Apr 03, 2024 at 08:38:17AM -0400, Stephen Smoogen wrote:
>  Can you give the full httpd string (remove the ip address). This
> might help
>  figure out if it is a specific version of a tool which is the problem or if
>  it is a generic issue. 
I don't log user agents, sorry. I could temporarily enable that if
helps?
-- 
Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

[CentOS-mirror] Abusive repodata downloads

[Stephen Smoogen]

On Wed, 3 Apr 2024 at 08:42, Carsten Otto <otto(a)informatik.rwth-aachen.de&gt;
wrote:

>  On Wed, Apr 03, 2024 at 08:38:17AM -0400, Stephen Smoogen wrote:
>  > Can you give the full httpd string (remove the ip address). This might
>  help
>  > figure out if it is a specific version of a tool which is the problem or
>  if
>  > it is a generic issue.
>
>  I don't log user agents, sorry. I could temporarily enable that if
>  helps?
>  
If you for debugging purposes and then turn it off again it would be
helpful. There are multiple tools which can download this data from a
'client' point of view and some of them are going to be 'buggy'.


>  --
>  Dr. Carsten Otto
>  http://verify.rwth-aachen.de/otto/
>  

-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren

[CentOS-mirror] Abusive repodata downloads

[Carsten Otto]

On Wed, Apr 03, 2024 at 08:45:57AM -0400, Stephen Smoogen wrote:
>  If you for debugging purposes and then turn it off again it would be
>  helpful. There are multiple tools which can download this data from a
>  'client' point of view and some of them are going to be 'buggy'. 
Downloads a few files about once a minute:

libdnf (Rocky Linux 8.8; generic; Linux.x86_64)

/fedora-epel/8/Everything/x86_64/repodata/repomd.xml
/fedora-epel/8/Everything/x86_64/repodata/0e32b3c58c14790acdb973f20a4b2207e24720222513b0bc63297a649a9884a8-comps-Everything.x86_64.xml
/fedora-epel/8/Everything/x86_64/repodata/cf47b564c8fa629069ef0f0836dda95fff05e077068eb3e9ed554c343581be84-prestodelta.xml.xz

/fedora-epel/8/Everything/x86_64/repodata/6f0dd08e0df456c5c28d4f776aa71b4bb6d0e4bce4e4bf3423260efdd4b7af2b-updateinfo.xml.bz2
/fedora-epel/8/Everything/x86_64/repodata/1f3bb463d1f38d18f6af974e8353b7e8ad7f368686a462c94d664541f00c7c39-primary.xml.gz

/fedora-epel/8/Everything/x86_64/repodata/20761194528ef5ba68ccab20ed7ab3f9303c81ebc562f9dca772a7f20db59dd8-filelists.xml.gz

Similar issue, different user:

libdnf (Rocky Linux 9.3; generic; Linux.x86_64)

/fedora-epel/9/Everything/x86_64/repodata/repomd.xml 
/fedora-epel/9/Everything/x86_64/repodata/8768391f4f5cb0daebfd4f6103704b325933733c458df8f261b1e47601f81afc-comps-Everything.x86_64.xml.gz

/fedora-epel/9/Everything/x86_64/repodata/881a09d92cdb4078eed6995d8e2c48304c4a36011b6fdc695bc19705f86ab2fe-updateinfo.xml.bz2

/fedora-epel/9/Everything/x86_64/repodata/a38ae61a2abfb200cdc4a796f5478c8bcbba4b6a222ab46e7893d9879760734a-primary.xml.gz

/fedora-epel/9/Everything/x86_64/repodata/b2cd153314a9aed419fa6adf682c85503be7b6045772bf6cfffef2a4f6fd19fb-filelists.xml.gz


This user downloads other files, also about once a minute (via HTTP 2.0):

urlgrabber/3.10 yum/3.4.3

/fedora-epel/7/x86_64/repodata/repomd.xml 
/fedora-epel/7/x86_64/repodata/b067b70cf45ef66ab62dc5661b802249f45756ef620528ccf9843d04ca6da83b-updateinfo.xml.bz2

/fedora-epel/7/x86_64/repodata/11ce2dc3bd977b1cbb9acccf51629f6d07ee9c191fecd960ee82df1d307ef3cb-primary.sqlite.bz2


I'll provide more examples as soon as they pour in.
-- 
Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

[CentOS-mirror] Abusive repodata downloads

[Carsten Otto]

Every few minutes:

libdnf (Red Hat Enterprise Linux 8.6; generic; Linux.x86_64)

/fedora-epel/8/Everything/x86_64/repodata/repomd.xml 
/fedora-epel/8/Everything/x86_64/repodata/0e32b3c58c14790acdb973f20a4b2207e24720222513b0bc63297a649a9884a8-comps-Everything.x86_64.xml

/fedora-epel/8/Everything/x86_64/repodata/cf47b564c8fa629069ef0f0836dda95fff05e077068eb3e9ed554c343581be84-prestodelta.xml.xz

/fedora-epel/8/Everything/x86_64/repodata/6f0dd08e0df456c5c28d4f776aa71b4bb6d0e4bce4e4bf3423260efdd4b7af2b-updateinfo.xml.bz2

/fedora-epel/8/Everything/x86_64/repodata/1f3bb463d1f38d18f6af974e8353b7e8ad7f368686a462c94d664541f00c7c39-primary.xml.gz

/fedora-epel/8/Everything/x86_64/repodata/20761194528ef5ba68ccab20ed7ab3f9303c81ebc562f9dca772a7f20db59dd8-filelists.xml.gz


More examples (similar patterns to those already shown):

libdnf (AlmaLinux 8.8; generic; Linux.x86_64)
libdnf (Rocky Linux 8.8; generic; Linux.x86_64)
libdnf (Rocky Linux 8.9; generic; Linux.x86_64)
libdnf (Rocky Linux 9.2; generic; Linux.x86_64)
urlgrabber/3.10 yum/3.4.3 (HTTP 2.0)
-- 
Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

[Dennis Gilmore]

On Tue, Apr 2, 2024 at 7:08 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  On Tue, Apr 02, 2024 at 12:35:53PM +0200, Carsten Otto wrote:
>  > Hi,
>
>  Hello.
>
>  > I'm one of the admins of ftp.halifax.rwth-aachen.de, which offers fedora
>  > and fedora-epel among other distributions and projects. I've recently
>  > noticed that many Fedora users (including EPEL, CentOS, and BlackArch)
>  > frequently re-download "repodata" files that haven't been updated
> since
>  > the previous request.
>
>  How are you determining this? From IP address?
>  Those could be a large number of different machines behind a nat proxy
>  no?
>
>  > As some of those files are rather large ("filelists") and others are hit
>  > extremely often, I added fail2ban rules to deny users access to our
>  > service for some time.
>
>  Recent versions of fedora do not even download the filelists unless a
>  user requests/looks up something that needs it. For normal install or
>  updates, it shouldn't be pulled. Can you tell what OS/versions this is
>  happening with?
>
>  In general clients should pull the repomd.xml file and check if it's
>  changed, if not, use their cached versions. 

An exception to this would be people building containers.
Theytypically remove the metadata at the end of each build. So do not
have it cached.  However, the described behaviour would seem to be
more widespread than some people building containers, which may
indicate some other reason that a group of people are repeatedly
grabbing the same data. It would be interesting to know if they are
only getting repodata or also downloading rpms and other things.

Dennis
> > Please adjust your software so that file mirrors like ours are not
> > (ab)used like this. Files that haven't been changed usually shouldn't
be
> > downloaded.
>
> This should normally already be the case, so I wonder what is happening
> here, could be a bug in some version/os or otherwise something
> unexpected.
>
> kevin
> --
> _______________________________________________
> Mirror-admin mailing list -- mirror-admin(a)lists.fedoraproject.org
> To unsubscribe send an email to mirror-admin-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Carsten Otto]

On Wed, Apr 03, 2024 at 12:54:10PM -0500, Dennis Gilmore wrote:
>  An exception to this would be people building containers.
>  Theytypically remove the metadata at the end of each build. So do not
>  have it cached.  However, the described behaviour would seem to be
>  more widespread than some people building containers, which may
>  indicate some other reason that a group of people are repeatedly
>  grabbing the same data. It would be interesting to know if they are
>  only getting repodata or also downloading rpms and other things. 
Interesting thought! I just grepped for a few of the "abusive" hosts.
These hosts are downloading packages, including some "basic" ones like
perl, curl, python setuptools.

I'll relax the fail2ban rules a bit and see how things evolve over time.
Maybe everyone needs to rebuild everything once now, which obviously
causes a bit of traffic.
-- 
Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

[Stephen John Smoogen]

On Wed, 3 Apr 2024 at 14:08, Carsten Otto <otto(a)informatik.rwth-aachen.de&gt;
wrote:

>  On Wed, Apr 03, 2024 at 12:54:10PM -0500, Dennis Gilmore wrote:
>  > An exception to this would be people building containers.
>  > Theytypically remove the metadata at the end of each build. So do not
>  > have it cached.  However, the described behaviour would seem to be
>  > more widespread than some people building containers, which may
>  > indicate some other reason that a group of people are repeatedly
>  > grabbing the same data. It would be interesting to know if they are
>  > only getting repodata or also downloading rpms and other things.
>
>  Interesting thought! I just grepped for a few of the "abusive" hosts.
>  These hosts are downloading packages, including some "basic" ones like
>  perl, curl, python setuptools.
>
>  I'll relax the fail2ban rules a bit and see how things evolve over time.
>  Maybe everyone needs to rebuild everything once now, which obviously
>  causes a bit of traffic.
>  
My expectation is that traffic is going to be higher in the next 6 months
as various systems are getting moved from CentOS 7 to some newer release.
This upgrade process will require a system to be first updated to the
latest '7' and many have had updates turned off for years.. then there will
be a download and update to EL8 or EL9 depending on what they are doing
which will also create a spike in traffic.



>  --
>  Dr. Carsten Otto
>  http://verify.rwth-aachen.de/otto/
>  --
>  _______________________________________________
>  Mirror-admin mailing list -- mirror-admin(a)lists.fedoraproject.org
>  To unsubscribe send an email to mirror-admin-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>  

-- 
Stephen J Smoogen.
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren

[Carsten Otto]

On Wed, Apr 03, 2024 at 02:14:07PM -0400, Stephen John Smoogen wrote:
>  My expectation is that traffic is going to be higher in the next 6
> months
>  as various systems are getting moved from CentOS 7 to some newer release.
>  This upgrade process will require a system to be first updated to the
>  latest '7' and many have had updates turned off for years.. then there will
>  be a download and update to EL8 or EL9 depending on what they are doing
>  which will also create a spike in traffic. 
Even better, that sounds like useful traffic and I have around 15
GBit/sec to spare for that :)
-- 
Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

[Starburst Services SysOp Team]

I'll just chime in with what we have been seeing.

We run mirror servers in multiple countries (US, UK, SG, AU), and ALL of 
the EPEL mirrors have been getting hit with DDoS Flood Attacks.

One of the Micro mirrors we had to shut down completely here in the US.



On 2024-04-02 10:35, Carsten Otto wrote:
>  Hi,
>
>  I'm one of the admins of ftp.halifax.rwth-aachen.de, which offers fedora
>  and fedora-epel among other distributions and projects. I've recently
>  noticed that many Fedora users (including EPEL, CentOS, and BlackArch)
>  frequently re-download "repodata" files that haven't been updated since
>  the previous request.
>
>  As some of those files are rather large ("filelists") and others are hit
>  extremely often, I added fail2ban rules to deny users access to our
>  service for some time.
>
>  Please adjust your software so that file mirrors like ours are not
>  (ab)used like this. Files that haven't been changed usually shouldn't be
>  downloaded.
>
>  Thanks
>  Carsten
>
>  --
>  _______________________________________________
>  Mirror-admin mailing list -- mirror-admin(a)lists.fedoraproject.org
>  To unsubscribe send an email to mirror-admin-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Paulson McIntyre]

Any chance some of the volume changes are due to patching/migrations
related to the recent SSH vulnerability?

-Paulson

On Wed, Apr 3, 2024 at 12:01 PM Starburst Services SysOp Team via
Mirror-admin <mirror-admin(a)lists.fedoraproject.org&gt; wrote:

>  I'll just chime in with what we have been seeing.
>
>  We run mirror servers in multiple countries (US, UK, SG, AU), and ALL of
>  the EPEL mirrors have been getting hit with DDoS Flood Attacks.
>
>  One of the Micro mirrors we had to shut down completely here in the US.
>
>
>
>  On 2024-04-02 10:35, Carsten Otto wrote:
>  > Hi,
>  >
>  > I'm one of the admins of ftp.halifax.rwth-aachen.de, which offers fedora
>  > and fedora-epel among other distributions and projects. I've recently
>  > noticed that many Fedora users (including EPEL, CentOS, and BlackArch)
>  > frequently re-download "repodata" files that haven't been updated
> since
>  > the previous request.
>  >
>  > As some of those files are rather large ("filelists") and others are hit
>  > extremely often, I added fail2ban rules to deny users access to our
>  > service for some time.
>  >
>  > Please adjust your software so that file mirrors like ours are not
>  > (ab)used like this. Files that haven't been changed usually shouldn't be
>  > downloaded.
>  >
>  > Thanks
>  > Carsten
>  >
>  > --
>  > _______________________________________________
>  > Mirror-admin mailing list -- mirror-admin(a)lists.fedoraproject.org
>  > To unsubscribe send an email to
>  mirror-admin-leave(a)lists.fedoraproject.org
>  > Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>  > Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>  --
>  _______________________________________________
>  Mirror-admin mailing list -- mirror-admin(a)lists.fedoraproject.org
>  To unsubscribe send an email to mirror-admin-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>

[Carsten Otto]

On Wed, Apr 03, 2024 at 12:16:56PM -0400, Paulson McIntyre wrote:
>  Any chance some of the volume changes are due to patching/migrations
>  related to the recent SSH vulnerability? 
I tried, but I wasn't able to find any outstanding file or host. It
looks like it's more traffic all over the board, which I guess happens
if lots of users need to update lots of packages.

I do have lots of old logs, do you know of a few specific files that I
could check?
-- 
Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

[Paulson McIntyre]

Interesting question....The repo metadata obviously, sshd, and xz/lzma
libs. Perhaps anything that depends on those three libs though.

-Paulson

On Wed, Apr 3, 2024 at 12:27 PM Carsten Otto <otto(a)informatik.rwth-aachen.de&gt;
wrote:

>  On Wed, Apr 03, 2024 at 12:16:56PM -0400, Paulson McIntyre wrote:
>  > Any chance some of the volume changes are due to patching/migrations
>  > related to the recent SSH vulnerability?
>
>  I tried, but I wasn't able to find any outstanding file or host. It
>  looks like it's more traffic all over the board, which I guess happens
>  if lots of users need to update lots of packages.
>
>  I do have lots of old logs, do you know of a few specific files that I
>  could check?
>  --
>  Dr. Carsten Otto
>  http://verify.rwth-aachen.de/otto/
>

[Carsten Otto]

On Wed, Apr 03, 2024 at 12:44:41PM -0400, Paulson McIntyre wrote:
>  Interesting question....The repo metadata obviously, sshd, and
> xz/lzma
>  libs. Perhaps anything that depends on those three libs though. 
I don't see any noteworthy change for fedora-epel downloads of files
including "ssh" (in March).
-- 
Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

[Paulson McIntyre]

>  No changes in sshd 
That's actually a bit scary given the
seriousness of the ssh vuln.

-Paulson

On Wed, Apr 3, 2024 at 1:10 PM Carsten Otto <otto(a)informatik.rwth-aachen.de&gt;
wrote:

>  On Wed, Apr 03, 2024 at 12:44:41PM -0400, Paulson McIntyre wrote:
>  > Interesting question....The repo metadata obviously, sshd, and xz/lzma
>  > libs. Perhaps anything that depends on those three libs though.
>
>  I don't see any noteworthy change for fedora-epel downloads of files
>  including "ssh" (in March).
>  --
>  Dr. Carsten Otto
>  http://verify.rwth-aachen.de/otto/
>

[Kevin Fenzi]

On Wed, Apr 03, 2024 at 01:20:31PM -0400, Paulson McIntyre wrote:
>  > No changes in sshd
>  That's actually a bit scary given the seriousness of the ssh vuln. 
RHEL/CentOS/Rocky/Alma/etc, never had the vulnerable version.
It was never even built for them. EPEL doesn't provide xz (thats in the
base EL repos).

If people are checking for updates, that could explain some increased
traffic, but I wouldn't think it would explain any dramatic traffic
increase. 

A number of other mirror admins don't seem to be seeing anything
different, so it sounds to me like it might be region specific perhaps?

Are all these requests ipv4? Are they coming from the same or different
providers?

I can try and look for patterns in the master mirror logs... 

kevin

[Carsten Otto]

On Wed, Apr 03, 2024 at 10:37:20AM -0700, Kevin Fenzi wrote:
>  RHEL/CentOS/Rocky/Alma/etc, never had the vulnerable version.
>  It was never even built for them. EPEL doesn't provide xz (thats in the
>  base EL repos). 
Ah, that explains the lack of actual downloads then :)

>  If people are checking for updates, that could explain some
> increased
>  traffic, but I wouldn't think it would explain any dramatic traffic
>  increase.  
It's about 10x, not dramatic, but still quite a lot - especially because
it's just metadata from the looks of it.

>  A number of other mirror admins don't seem to be seeing anything
>  different, so it sounds to me like it might be region specific perhaps?
>  
>  Are all these requests ipv4? Are they coming from the same or different
>  providers? 
As far as I can tell, it's all over the place. A few hosts are outliers
(maybe due to NAT or impatience), but overall, it's just more of
everything.

So, to summarize, the idea is that due to the xz/ssh issue, lots of
users are downloading metadata, especially the "filelists", causing an
ongoing surge in traffic.

If that's true, things should calm down anytime now.
-- 
Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

[Carsten Otto]

OK, I dug in deeper, a few more insights.

end of February / early March:
- 30k-40k hits for /fedora-epel/8/Everything/x86_64/repodata/repomd.xml,
  comps-Everything.x86_64.xml, -filelists.xml.gz, ...
- 10k hits or so for the same in /7/

end of March / early April:
- around 30k hits for /8/ and /9/ files
- 400k-700k (!) for /7/

Top hitters:
/fedora-epel/7/x86_64/repodata/37f85d91d087d574c4551a96731a6e8f2d6b2a458d364901fdeb7858e6fc7950-updateinfo.xml.bz2
/fedora-epel/7/x86_64/repodata/37f85d91d087d574c4551a96731a6e8f2d6b2a458d364901fdeb7858e6fc7950-updateinfo.xml.bz2
/fedora-epel/7/x86_64/repodata/6111d049c4d53fc00176b3da63856b580c6a34f1b7a2062ccf821988d095ec44-updateinfo.xml.bz2
/fedora-epel/7/x86_64/repodata/6fa65768495a2a46eff12c510acc23d4df39a85df14958ac5bfea0f758d9a43a-primary.sqlite.bz2
/fedora-epel/7/x86_64/repodata/e52552f46d9f450b7b5966d4cdafdc58769eda686fc2d9a4c44beb441b122453-primary.sqlite.bz2
/fedora-epel/7/x86_64/repodata/e52552f46d9f450b7b5966d4cdafdc58769eda686fc2d9a4c44beb441b122453-primary.sqlite.bz2
/fedora-epel/7/x86_64/repodata/repomd.xml
/fedora-epel/7/x86_64/repodata/repomd.xml
/fedora-epel/7/x86_64/repodata/repomd.xml

It looks like the number of requests for the /7/ files picked up during
March, but a sudden increase happened around March 28th.

It looks like traffic jumped up due to downloads of "-filelists.xml" files
at that time, but I also see lots of traffic for "-primary.sqlite" and
"-updateinfo". 

That's about the time when the libxz mess become public.

When did you start shipping security updates to SSH/liblzma/...?
-- 
Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

[Carsten Otto]

Hi again,

yesterday something happened, the fedora-epel load on our mirror
suddenly dropped to almost zero - see attached images.

I don't know what's going on.

Bye
Carsten
-- 
Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

[Adrian Reber]

On Fri, Apr 12, 2024 at 08:29:59AM +0200, Carsten Otto wrote:
>  yesterday something happened, the fedora-epel load on our mirror
>  suddenly dropped to almost zero - see attached images. 
Your mirror was not up to date and was dropped from the list of up to
date mirrors, the crawler already detected that your mirror is up to
date again and in about 30 minutes it should see traffic again.

		Adrian

[Carsten Otto]

On Fri, Apr 12, 2024 at 08:51:28AM +0200, Adrian Reber wrote:
>  Your mirror was not up to date and was dropped from the list of up
> to
>  date mirrors, the crawler already detected that your mirror is up to
>  date again and in about 30 minutes it should see traffic again. 
Oh, interesting. Should I update more frequently than every two hours?
Do you offer SSH push triggers / rsync pushes?

Thanks
Carsten
-- 
Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

[Adrian Reber]

On Fri, Apr 12, 2024 at 09:08:37AM +0200, Carsten Otto wrote:
>  On Fri, Apr 12, 2024 at 08:51:28AM +0200, Adrian Reber wrote:
>  > Your mirror was not up to date and was dropped from the list of up to
>  > date mirrors, the crawler already detected that your mirror is up to
>  > date again and in about 30 minutes it should see traffic again.
>  
>  Oh, interesting. Should I update more frequently than every two hours? 
It depends. If you are using quick-fedora-mirror you can do it every 10
minutes. Using full rsync would result in higher I/O load. MirrorManager
scans EPEL mirrors every 6 hours, so there is always the possibility to
be removed from the list of active mirrors.

With the help of quick-fedora-mirror I am running my mirror scripts
every hour, but it takes between 2 and 5 seconds to decide if the
primary mirror has changed. Just using rsync would probably require more
time. So I think every two hours sounds right.

>  Do you offer SSH push triggers / rsync pushes? 
No.

		Adrian

[Jasper Yu]

Hi,

I want to raise a new issue here, is the origin server has TCP settings
such as BBR and buffer settings.
If not, is it possible to set them up?

It is quite slow to sync the contents when there is high latency.
Our solution ends up
- open an VPS near the origin server
- configure with those setting
- have a TCP proxy using socat
and which makes us the only server in north east asia still updated during
that time.

We also observe traffic increase and those traffic are coming from other
countries.
[image: image.png]
[image: image.png]

Thanks
ᐧ

On Fri, Apr 12, 2024 at 5:55 PM Adrian Reber via Mirror-admin <
mirror-admin(a)lists.fedoraproject.org&gt; wrote:

>  On Fri, Apr 12, 2024 at 09:08:37AM +0200, Carsten Otto wrote:
>  > On Fri, Apr 12, 2024 at 08:51:28AM +0200, Adrian Reber wrote:
>  > > Your mirror was not up to date and was dropped from the list of up to
>  > > date mirrors, the crawler already detected that your mirror is up to
>  > > date again and in about 30 minutes it should see traffic again.
>  >
>  > Oh, interesting. Should I update more frequently than every two hours?
>
>  It depends. If you are using quick-fedora-mirror you can do it every 10
>  minutes. Using full rsync would result in higher I/O load. MirrorManager
>  scans EPEL mirrors every 6 hours, so there is always the possibility to
>  be removed from the list of active mirrors.
>
>  With the help of quick-fedora-mirror I am running my mirror scripts
>  every hour, but it takes between 2 and 5 seconds to decide if the
>  primary mirror has changed. Just using rsync would probably require more
>  time. So I think every two hours sounds right.
>
>  > Do you offer SSH push triggers / rsync pushes?
>
>  No.
>
>                  Adrian
>  --
>  _______________________________________________
>  Mirror-admin mailing list -- mirror-admin(a)lists.fedoraproject.org
>  To unsubscribe send an email to mirror-admin-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>  

-- 
Jasper Yu
About me: https://seadog007.me
CV: https://seadog007.work
Email: 007seadog(a)gmail.com, admin(a)seadog007.me
Phone: [phone]

[Adrian Reber]

On Fri, Apr 12, 2024 at 09:05:24PM +0800, Jasper Yu wrote:
>  We also observe traffic increase and those traffic are coming from
> other
>  countries. 
Unfortunately I don't know where your mirror is located. I also do not
know the timezone on your diagrams and what is causing the traffic
exactly, but if you look at the timestamps of EPEL you can see that
today around 3am (UTC) a new compose was created and pushed to the
mirrors, that will result in more clients downloading the changed
metadata and potentially the corresponding updates from mirrors. If
there are not many mirror in your region this can result in a lot of
traffic.

		Adrian