9:08 a.m.
Hello,
I've noticed our rsyncs of fedora sometimes end up crawling along at
around 50-150kB/s. When there's a lot of content to mirror, this means
the sync could take several days, or even a week - although I'd imagine
the remote end would terminate the connection long before that.
In normal circumstances, we'd get anything between 5-10MB/s from
dl.fedoraproject.org. I was thinking that a particular host in pool of
servers was having an issue, so I did an rsync from it directly,
however the speed was normal:
rsync -Pav \
209.132.181.26::fedora-buffet0/fedora/linux/releases/24/Workstation/x86_64/iso/Fedora-Workstation-netinst-x86_64-24-1.2.iso
.
---8<---
receiving incremental file list
Fedora-Workstation-netinst-x86_64-24-1.2.iso
460324864 100% 6.61MB/s 0:01:06 (xfer#1, to-check=0/1)
sent 55 bytes received 460381543 bytes 6720899.24 bytes/sec
total size is 460324864 speedup is 1.00
These slow downs only seem to occur when we use dl.fedoraproject.org,
if we switch to another host such as download-i2.fedoraproject.org, for
example, it remains stable.
dl.fedoraproject.org is around 150ms away from us (connecting from
Dublin, Ireland), but I wouldn't imagine this would cause such drastic
speed drops.
rg
--
Rob Gallagher | Public Key: 0x841386E3
HEAnet Limited, Ireland's Education and Research Network
1st Floor, 5 George's Dock, IFSC, Dublin D01 X8N7
Registered in Ireland, no 275301
☎ (+353-1) 6609040 WWW: http://www.heanet.ie/
12:17 p.m.
On 25 August 2016 at 10:08, Rob Gallagher <robert.gallagher(a)heanet.ie> wrote:
Hello,
I've noticed our rsyncs of fedora sometimes end up crawling along at
around 50-150kB/s. When there's a lot of content to mirror, this means
the sync could take several days, or even a week - although I'd imagine
the remote end would terminate the connection long before that.
In normal circumstances, we'd get anything between 5-10MB/s from
dl.fedoraproject.org. I was thinking that a particular host in pool of
servers was having an issue, so I did an rsync from it directly,
however the speed was normal:
OK the load average on the servers is a bit up but the NFS throughput
looks about normal.. so I am not sure what is causing it yet. I will
investigate more on my side.
Could you look on your side for the following:
1) Can you give us an mtr from your host to dl05.fedoraproject.org (it
is one of the dl servers). Run for about 10 minutes.
2) Do this again while doing an rsync. Send us both at the end of those runs.
3) Can you try doing an rsync from the various dl servers to see if
there is one which is slow
rsync -Pav \
209.132.181.26::fedora-buffet0/fedora/linux/releases/24/Workstation/x86_64/iso/Fedora-Workstation-netinst-x86_64-24-1.2.iso
.
---8<---
receiving incremental file list
Fedora-Workstation-netinst-x86_64-24-1.2.iso
460324864 100% 6.61MB/s 0:01:06 (xfer#1, to-check=0/1)
sent 55 bytes received 460381543 bytes 6720899.24 bytes/sec
total size is 460324864 speedup is 1.00
These slow downs only seem to occur when we use dl.fedoraproject.org,
if we switch to another host such as download-i2.fedoraproject.org, for
example, it remains stable.
dl.fedoraproject.org is around 150ms away from us (connecting from
Dublin, Ireland), but I wouldn't imagine this would cause such drastic
speed drops.
ms round trip time has little bearing anymore in bandwidth issues :(.
You can have an mid-stream which is throttling you down for some
reason (they don't think the next guy is paying them enough) or you
can have a bad cable somewhere which pings perfectly nicely but stops
up.
--
Stephen J Smoogen.
6:21 a.m.
Hi Stephen,
On Thu, 25 Aug 2016 13:17:33 -0400
Stephen John Smoogen <smooge(a)gmail.com> wrote:
OK the load average on the servers is a bit up but the NFS
throughput
looks about normal.. so I am not sure what is causing it yet. I will
investigate more on my side.
Could you look on your side for the following:
1) Can you give us an mtr from your host to dl05.fedoraproject.org (it
is one of the dl servers). Run for about 10 minutes.
2) Do this again while doing an rsync. Send us both at the end of
those runs. 3) Can you try doing an rsync from the various dl servers
to see if there is one which is slow
Thanks for looking into it, I'm attaching an mtr to dl05 with a 10
minute run, along with one when rsync is running.
A log of rsyncs from the 5 servers in the dl.fedoraproject.org rotation
is also attached.
ms round trip time has little bearing anymore in bandwidth issues :(.
You can have an mid-stream which is throttling you down for some
reason (they don't think the next guy is paying them enough) or you
can have a bad cable somewhere which pings perfectly nicely but stops
up.
Yeah I didn't think so, we provide internet access for the majority of
academic institutions in Ireland, so our connectivity is usually pretty
good :)
rg
--
Rob Gallagher | Public Key: 0x841386E3
HEAnet Limited, Ireland's Education and Research Network
1st Floor, 5 George's Dock, IFSC, Dublin D01 X8N7
Registered in Ireland, no 275301
☎ (+353-1) 6609040 WWW: http://www.heanet.ie/
9:43 a.m.
On 26 August 2016 at 07:21, Rob Gallagher <robert.gallagher(a)heanet.ie> wrote:
Hi Stephen,
On Thu, 25 Aug 2016 13:17:33 -0400
Stephen John Smoogen <smooge(a)gmail.com> wrote:
> OK the load average on the servers is a bit up but the NFS throughput
> looks about normal.. so I am not sure what is causing it yet. I will
> investigate more on my side.
>
> Could you look on your side for the following:
> 1) Can you give us an mtr from your host to dl05.fedoraproject.org (it
> is one of the dl servers). Run for about 10 minutes.
> 2) Do this again while doing an rsync. Send us both at the end of
> those runs. 3) Can you try doing an rsync from the various dl servers
> to see if there is one which is slow
>
Thanks for looking into it, I'm attaching an mtr to dl05 with a 10
minute run, along with one when rsync is running.
I like how to get to the UK from ireland it takes 180 ms but it takes
150ms to get to the US after going through there.. ICMP flow control
is wonderful :). (Or Brexit affects packets.. take your pick).
A log of rsyncs from the 5 servers in the dl.fedoraproject.org
rotation
is also attached.
OK dl01 and dl02 seem to be the problem ones in your check. I will
look them over and see what is up. Could you try dl04 for a bit and
see if you have a better set of runs?
>
> ms round trip time has little bearing anymore in bandwidth issues :(.
> You can have an mid-stream which is throttling you down for some
> reason (they don't think the next guy is paying them enough) or you
> can have a bad cable somewhere which pings perfectly nicely but stops
> up.
Yeah I didn't think so, we provide internet access for the majority of
academic institutions in Ireland, so our connectivity is usually pretty
good :)
rg
--
Rob Gallagher | Public Key: 0x841386E3
HEAnet Limited, Ireland's Education and Research Network
1st Floor, 5 George's Dock, IFSC, Dublin D01 X8N7
Registered in Ireland, no 275301
☎ (+353-1) 6609040 WWW: http://www.heanet.ie/
--
Stephen J Smoogen.
10:08 a.m.
Hi Stephen,
There's actually an rsync running at the moment which is exhibiting the
slowness issue, the sync is happening from from dl03 (209.132.181.25),
and it's showing speeds around 100kB/s. We're connecting from
193.1.219.88.
rg
On Fri, 26 Aug 2016 10:43:14 -0400
Stephen John Smoogen <smooge(a)gmail.com> wrote:
On 26 August 2016 at 07:21, Rob Gallagher
<robert.gallagher(a)heanet.ie> wrote:
> Hi Stephen,
>
> On Thu, 25 Aug 2016 13:17:33 -0400
> Stephen John Smoogen <smooge(a)gmail.com> wrote:
>
>> OK the load average on the servers is a bit up but the NFS
>> throughput looks about normal.. so I am not sure what is causing
>> it yet. I will investigate more on my side.
>>
>> Could you look on your side for the following:
>> 1) Can you give us an mtr from your host to dl05.fedoraproject.org
>> (it is one of the dl servers). Run for about 10 minutes.
>> 2) Do this again while doing an rsync. Send us both at the end of
>> those runs. 3) Can you try doing an rsync from the various dl
>> servers to see if there is one which is slow
>>
>
> Thanks for looking into it, I'm attaching an mtr to dl05 with a 10
> minute run, along with one when rsync is running.
>
I like how to get to the UK from ireland it takes 180 ms but it takes
150ms to get to the US after going through there.. ICMP flow control
is wonderful :). (Or Brexit affects packets.. take your pick).
> A log of rsyncs from the 5 servers in the dl.fedoraproject.org
> rotation is also attached.
>
OK dl01 and dl02 seem to be the problem ones in your check. I will
look them over and see what is up. Could you try dl04 for a bit and
see if you have a better set of runs?
>>
>> ms round trip time has little bearing anymore in bandwidth
>> issues :(. You can have an mid-stream which is throttling you down
>> for some reason (they don't think the next guy is paying them
>> enough) or you can have a bad cable somewhere which pings
>> perfectly nicely but stops up.
>
> Yeah I didn't think so, we provide internet access for the majority
> of academic institutions in Ireland, so our connectivity is usually
> pretty good :)
>
> rg
>
> --
> Rob Gallagher | Public Key: 0x841386E3
>
> HEAnet Limited, Ireland's Education and Research Network
> 1st Floor, 5 George's Dock, IFSC, Dublin D01 X8N7
> Registered in Ireland, no 275301
> ☎ (+353-1) 6609040 WWW: http://www.heanet.ie/
>
--
Rob Gallagher | Public Key: 0x841386E3
HEAnet Limited, Ireland's Education and Research Network
1st Floor, 5 George's Dock, IFSC, Dublin D01 X8N7
Registered in Ireland, no 275301
☎ (+353-1) 6609040 WWW: http://www.heanet.ie/
10:28 a.m.
Thanks. I have looked and the server is running fine and could do
another rsync with no trouble. I did see that the route to get to your
server is a completely different one from what you included:
7. 209.132.181.98
0.0% 101
4.2 1.1 0.7 4.6 0.8
8. 209.132.181.163
0.0% 101
0.7 2.6 0.6 21.3 3.9
9. te0-0-1-0.nr11.b019174-0.phx02.atlas.cogentco.com
0.0% 101
5.7 1.6 1.1 5.7 0.7
10. 154.24.18.25
0.0% 101
2.0 2.8 1.8 8.3 1.2
11. te0-4-0-5.ccr21.phx02.atlas.cogentco.com
0.0% 101
2.0 3.1 1.8 8.9 1.4
12. be2931.ccr21.lax01.atlas.cogentco.com
0.0% 101
15.6 14.5 13.2 21.4 1.4
13. be2964.ccr41.lax04.atlas.cogentco.com
0.0% 101
13.6 14.5 13.4 19.3 1.2
14. level3.lax04.atlas.cogentco.com
0.0% 101
14.0 15.0 13.0 52.7 5.3
15. ae-3-3.car1.Dublin1.Level3.net
0.0% 101
218.7 212.2 180.3 399.2 62.0
16. HEANET-LIMI.car1.Dublin.Level3.net
0.0% 101
179.2 177.5 175.5 180.6 1.1
17. te5-1-blanch-sr1.services.hea.net
0.0% 101
176.2 177.4 176.2 183.2 1.6
18. rsyncer.ftp.heanet.ie
1.0% 101
176.0 176.3 175.9 180.3 0.6
So do we have mismatched routing going on?
On 26 August 2016 at 11:08, Rob Gallagher <robert.gallagher(a)heanet.ie> wrote:
Hi Stephen,
There's actually an rsync running at the moment which is exhibiting the
slowness issue, the sync is happening from from dl03 (209.132.181.25),
and it's showing speeds around 100kB/s. We're connecting from
193.1.219.88.
rg
On Fri, 26 Aug 2016 10:43:14 -0400
Stephen John Smoogen <smooge(a)gmail.com> wrote:
> On 26 August 2016 at 07:21, Rob Gallagher
> <robert.gallagher(a)heanet.ie> wrote:
> > Hi Stephen,
> >
> > On Thu, 25 Aug 2016 13:17:33 -0400
> > Stephen John Smoogen <smooge(a)gmail.com> wrote:
> >
> >> OK the load average on the servers is a bit up but the NFS
> >> throughput looks about normal.. so I am not sure what is causing
> >> it yet. I will investigate more on my side.
> >>
> >> Could you look on your side for the following:
> >> 1) Can you give us an mtr from your host to dl05.fedoraproject.org
> >> (it is one of the dl servers). Run for about 10 minutes.
> >> 2) Do this again while doing an rsync. Send us both at the end of
> >> those runs. 3) Can you try doing an rsync from the various dl
> >> servers to see if there is one which is slow
> >>
> >
> > Thanks for looking into it, I'm attaching an mtr to dl05 with a 10
> > minute run, along with one when rsync is running.
> >
>
> I like how to get to the UK from ireland it takes 180 ms but it takes
> 150ms to get to the US after going through there.. ICMP flow control
> is wonderful :). (Or Brexit affects packets.. take your pick).
>
> > A log of rsyncs from the 5 servers in the dl.fedoraproject.org
> > rotation is also attached.
> >
>
> OK dl01 and dl02 seem to be the problem ones in your check. I will
> look them over and see what is up. Could you try dl04 for a bit and
> see if you have a better set of runs?
>
> >>
> >> ms round trip time has little bearing anymore in bandwidth
> >> issues :(. You can have an mid-stream which is throttling you down
> >> for some reason (they don't think the next guy is paying them
> >> enough) or you can have a bad cable somewhere which pings
> >> perfectly nicely but stops up.
> >
> > Yeah I didn't think so, we provide internet access for the majority
> > of academic institutions in Ireland, so our connectivity is usually
> > pretty good :)
> >
> > rg
> >
> > --
> > Rob Gallagher | Public Key: 0x841386E3
> >
> > HEAnet Limited, Ireland's Education and Research Network
> > 1st Floor, 5 George's Dock, IFSC, Dublin D01 X8N7
> > Registered in Ireland, no 275301
> > ☎ (+353-1) 6609040 WWW: http://www.heanet.ie/
> >
>
>
>
--
Rob Gallagher | Public Key: 0x841386E3
HEAnet Limited, Ireland's Education and Research Network
1st Floor, 5 George's Dock, IFSC, Dublin D01 X8N7
Registered in Ireland, no 275301
☎ (+353-1) 6609040 WWW: http://www.heanet.ie/
--
Stephen J Smoogen.
12:02 p.m.
On 26 August 2016 at 11:28, Stephen John Smoogen <smooge(a)gmail.com> wrote:
Thanks. I have looked and the server is running fine and could do
another rsync with no trouble. I did see that the route to get to your
server is a completely different one from what you included:
Talked with our networking and the asymmetric routing is expected but
the route back must be overly congested. They are opening a ticket
with the level 2 network provider to see what can be done about it. If
other people are seeing this problem in Europe please send me the ip
address of your server so I can get routes back and see what is up.
--
Stephen J Smoogen.
2825
days inactive
2826
days old
mirror-admin@lists.fedoraproject.org
6 comments
2 participants
participants (2)
-
Rob Gallagher -
Stephen John Smoogen