infrastructure problem: koji and bodhi responding very slow / dl with 80kb/s

[Marius Schwarz]

Hi,

while testing ff 73 update, i noticed that bodhi and koji are sponding
very slow. The download speed at koji for germany was around 80KB/s,
while other parts of the country reached MB/s easily. 

As bodhi does not have that much html to transfer, i don't think it's a
network issue.

Could someone from the infrastructure team take a look at your dashboards?


best regards,
Marius

[Kevin Fenzi]

On Thu, Feb 13, 2020 at 03:31:26PM +0100, Marius Schwarz wrote:
>  
>  Hi,
>  
>  while testing ff 73 update, i noticed that bodhi and koji are sponding
>  very slow. The download speed at koji for germany was around 80KB/s,
>  while other parts of the country reached MB/s easily. 
>  
>  As bodhi does not have that much html to transfer, i don't think it's a
>  network issue. 
I haven't seen any other reports and it's very fast here. 

Do note that koji and bodhi are both things that are in one place: our
main datacenter in phx2 (arizona, usa). They aren't distributed. 
So, I suspect it _is_ network between there and you.

Unfortunately, I am not sure there's too much we can do about it. 
If it's some upstream network issue we can try and talk to providers,
but we seldom have any luck getting them to do anything. 

We are going to be moving from the phx2 datacenter to another facility
in virgina, usa later this year. I have no idea if that will improve
things for you though. ;( 

For koji downloads (those things that use kojipkgs.fedoraproject.org) we
might be able to setup amazon cloudfront, that would help with those. 

>  Could someone from the infrastructure team take a look at your
> dashboards? 
Do please note that if you hit an infrastructure problem, it's really
best to open a infrastructure ticket, not post to devel list (which we
read in our spare time): 

https://pagure.io/fedora-infrastructure/issues

Sorry I couldn't be of more help...

kevin

[Robert-André Mauchin]

On Thursday, 13 February 2020 18:18:03 CET Kevin Fenzi wrote:
>  On Thu, Feb 13, 2020 at 03:31:26PM +0100, Marius Schwarz wrote:
>  > Hi,
>  > 
>  > while testing ff 73 update, i noticed that bodhi and koji are sponding
>  > very slow. The download speed at koji for germany was around 80KB/s,
>  > while other parts of the country reached MB/s easily. 
>  > 
>  > As bodhi does not have that much html to transfer, i don't think it's a
>  > network issue.
>  
>  I haven't seen any other reports and it's very fast here.
>  
>  Do note that koji and bodhi are both things that are in one place: our
>  main datacenter in phx2 (arizona, usa). They aren't distributed.
>  So, I suspect it _is_ network between there and you.
>  
>  Unfortunately, I am not sure there's too much we can do about it.
>  If it's some upstream network issue we can try and talk to providers,
>  but we seldom have any luck getting them to do anything.
>  
>  We are going to be moving from the phx2 datacenter to another facility
>  in virgina, usa later this year. I have no idea if that will improve
>  things for you though. ;(
>  
>  For koji downloads (those things that use kojipkgs.fedoraproject.org) we
>  might be able to setup amazon cloudfront, that would help with those.
>  
>  > Could someone from the infrastructure team take a look at your dashboards?
>  
>  Do please note that if you hit an infrastructure problem, it's really
>  best to open a infrastructure ticket, not post to devel list (which we
>  read in our spare time):
>  
>  https://pagure.io/fedora-infrastructure/issues
>  
>  Sorry I couldn't be of more help...
>  
>  kevin 
I don't have any issue with Bodhi, but Koji is extremely slow for me too, it 
isn't a bandwidth problem but the time the server takes to answer a page 
request is always several minutes, which makes it excruciatingly annoying when 
trying to get logs for a failed build.

Best regards,

Robert-André

[Kevin Fenzi]

On Thu, Feb 13, 2020 at 11:46:00PM +0100, Robert-André Mauchin wrote:
>  I don't have any issue with Bodhi, but Koji is extremely slow for
> me too, it 
>  isn't a bandwidth problem but the time the server takes to answer a page 
>  request is always several minutes, which makes it excruciatingly annoying when 
>  trying to get logs for a failed build. 
Feel free to open a ticket if you can get us exact pages / load times? 

Perhaps it's something specific to the packages? 

My failed builds load pretty quickly here... 

kevin

[Artem Tim]

Same here. I am waiting several minutes for every my action. This is very unproductive.

[Leigh Griffin]

On Fri, Feb 14, 2020 at 12:59 AM Artem Tim <ego.cordatus(a)gmail.com&gt; wrote:

>  Same here. I am waiting several minutes for every my action. This is
> very
>  unproductive.
>  
Log it as a ticket and I can get the CPE folks to investigate it and see if
there is an obvious problem that we can help with.

>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  

-- 

Leigh Griffin

Engineering Manager

Red Hat Waterford <https://www.redhat.com/>

Communications House

Cork Road, Waterford City

lgriffin(a)redhat.com
M: [phone]     IM: lgriffin
@redhatjobs <https://twitter.com/redhatjobs>   redhatjobs
<https://www.facebook.com/redhatjobs> @redhatjobs
<https://instagram.com/redhatjobs>
<https://red.ht/sig>

[Stephen John Smoogen]

On Fri, 14 Feb 2020 at 06:21, Leigh Griffin <lgriffin(a)redhat.com&gt; wrote:

>  On Fri, Feb 14, 2020 at 12:59 AM Artem Tim <ego.cordatus(a)gmail.com&gt; wrote:
>
> > Same here. I am waiting several minutes for every my action. This is very
> > unproductive.
> >
>
>  Log it as a ticket and I can get the CPE folks to investigate it and see
>  if there is an obvious problem that we can help with.
>  

for people having this issue the ticket is
https://pagure.io/fedora-infrastructure/issue/8648

We need an idea of the geolocation you are coming from, traceroutes of
where you are going from and if possible in the browser any links etc which
are showing very slow (aka in Firefox right click inspect, and switch to
the network page and do a ^R on the koji.fedoraproject.org page)




>  _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedoraproject.org/archives/list/[e-mail]
> >
>
>
>  --
>
>  Leigh Griffin
>
>  Engineering Manager
>
>  Red Hat Waterford <https://www.redhat.com/>
>
>  Communications House
>
>  Cork Road, Waterford City
>
>  lgriffin(a)redhat.com
>  M: [phone]     IM: lgriffin
>  @redhatjobs <https://twitter.com/redhatjobs>   redhatjobs
>  <https://www.facebook.com/redhatjobs> @redhatjobs
>  <https://instagram.com/redhatjobs>
>  <https://red.ht/sig>
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  

-- 
Stephen J Smoogen.

[Kevin Fenzi]

On Fri, Feb 14, 2020 at 08:07:19AM -0500, Stephen John Smoogen wrote:
>  On Fri, 14 Feb 2020 at 06:21, Leigh Griffin
> <lgriffin(a)redhat.com&gt; wrote:
>  
>  >
>  >
>  > On Fri, Feb 14, 2020 at 12:59 AM Artem Tim <ego.cordatus(a)gmail.com&gt; wrote:
>  >
>  >> Same here. I am waiting several minutes for every my action. This is very
>  >> unproductive.
>  >>
>  >
>  > Log it as a ticket and I can get the CPE folks to investigate it and see
>  > if there is an obvious problem that we can help with.
>  >
>  
>  
>  for people having this issue the ticket is
>  https://pagure.io/fedora-infrastructure/issue/8648
>  
>  We need an idea of the geolocation you are coming from, traceroutes of
>  where you are going from and if possible in the browser any links etc which
>  are showing very slow (aka in Firefox right click inspect, and switch to
>  the network page and do a ^R on the koji.fedoraproject.org page) 
Also, it might help if people noted the exact time of day (UTC) they are
accessing things?

I suspect it may be backup slowness (which happens after 00:00UTC +-
2hours)

kevin

[Marius Schwarz]

Am 14.02.20 um 17:57 schrieb Kevin Fenzi:
>  Also, it might help if people noted the exact time of day (UTC) they are
>  accessing things?
>
>  I suspect it may be backup slowness (which happens after 00:00UTC +-
>  2hours)
>  
it was around 12AM CET (+100), so the backup is off the hook.

unfortune for us firefox just said "yesterday" when i asked him for the
time :D

best regards,
Marius

[Marius Schwarz]

Am 14.02.20 um 17:57 schrieb Kevin Fenzi:
>  On Fri, Feb 14, 2020 at 08:07:19AM -0500, Stephen John Smoogen
> wrote:
> > On Fri, 14 Feb 2020 at 06:21, Leigh Griffin <lgriffin(a)redhat.com&gt; wrote:
> >
> >>
> >> On Fri, Feb 14, 2020 at 12:59 AM Artem Tim <ego.cordatus(a)gmail.com&gt; wrote:
> >>
> >>> Same here. I am waiting several minutes for every my action. This is very
> >>> unproductive.
> >>>
> >> Log it as a ticket and I can get the CPE folks to investigate it and see
> >> if there is an obvious problem that we can help with.
> >>
> >
> > for people having this issue the ticket is
> > https://pagure.io/fedora-infrastructure/issue/8648
> >
> > We need an idea of the geolocation you are coming from, traceroutes of
> > where you are going from and if possible in the browser any links etc which
> > are showing very slow (aka in Firefox right click inspect, and switch to
> > the network page and do a ^R on the koji.fedoraproject.org page)
>  Also, it might help if people noted the exact time of day (UTC) they are
>  accessing things?
>
>  I suspect it may be backup slowness (which happens after 00:00UTC +-
>  2hour) 
"NOW" aka. 14.2.2020 18:15 CET  the same pagecall to koji and firefox
rpm download where as smooth as they could be.

But, the koji test page from the ticket, had 4,5 seconds loadtime, which
was the first page i visited  in koji today.

I know this behaviour from my serverfarm, if the initial DNS resolver is
not responding and the timeout until the second is asked, kicks in.
As the server does no longer need to ask for an ip resolv on the second
visit (here firefox), it responds faster.

This does not explain the worst case download rate from yesterday, but
it can explain the results people have writte to the ticket for visiting
koji .

Best regards,
Marius

[Kevin Fenzi]

On Fri, Feb 14, 2020 at 06:21:34PM +0100, Marius Schwarz wrote:
>  Am 14.02.20 um 17:57 schrieb Kevin Fenzi:
>  > On Fri, Feb 14, 2020 at 08:07:19AM -0500, Stephen John Smoogen wrote:
>  >> On Fri, 14 Feb 2020 at 06:21, Leigh Griffin <lgriffin(a)redhat.com&gt; wrote:
>  >>
>  >>>
>  >>> On Fri, Feb 14, 2020 at 12:59 AM Artem Tim <ego.cordatus(a)gmail.com&gt;
> wrote:
>  >>>
>  >>>> Same here. I am waiting several minutes for every my action. This is
> very
>  >>>> unproductive.
>  >>>>
>  >>> Log it as a ticket and I can get the CPE folks to investigate it and see
>  >>> if there is an obvious problem that we can help with.
>  >>>
>  >>
>  >> for people having this issue the ticket is
>  >> https://pagure.io/fedora-infrastructure/issue/8648
>  >>
>  >> We need an idea of the geolocation you are coming from, traceroutes of
>  >> where you are going from and if possible in the browser any links etc which
>  >> are showing very slow (aka in Firefox right click inspect, and switch to
>  >> the network page and do a ^R on the koji.fedoraproject.org page)
>  > Also, it might help if people noted the exact time of day (UTC) they are
>  > accessing things?
>  >
>  > I suspect it may be backup slowness (which happens after 00:00UTC +-
>  > 2hour)
>  
>  "NOW" aka. 14.2.2020 18:15 CET  the same pagecall to koji and firefox
>  rpm download where as smooth as they could be. 
ok. So that means the task link loaded in under 4 seconds?
>  
>  But, the koji test page from the ticket, had 4,5 seconds loadtime, which
>  was the first page i visited  in koji today. 
Had before, but doesn't now? Or still does now?

>  I know this behaviour from my serverfarm, if the initial DNS resolver
> is
>  not responding and the timeout until the second is asked, kicks in.
>  As the server does no longer need to ask for an ip resolv on the second
>  visit (here firefox), it responds faster. 
You mean the end user dns? Sure, could cause problems indeed... but
thats out of our control.
>  
>  This does not explain the worst case download rate from yesterday, but
>  it can explain the results people have writte to the ticket for visiting
>  koji . 
It would seem odd for a variety of people to all hit dns issues at the
same time with different end user servers. 

Anyhow, I have asked for more info in the ticket, hopefully we can
figure something out. 

kevin