koji down?

[Richard Shaw]

I didn't see any other threads but I can't have been the first to notice...

I can't access the koji web interface or initiate builds...

Kerberos authentication fails: unable to obtain a session (gssapi auth
failed: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='
koji.fedoraproject.org', port=443): Read timed out. (read timeout=60))

Anyone else seeing this?

Thanks,
Richard

[Richard W.M. Jones]

On Wed, Nov 16, 2022 at 07:19:37AM -0600, Richard Shaw wrote:
> I didn't see any other threads but I can't have been the first to notice...
>
> I can't access the koji web interface or initiate builds...
>
> Kerberos authentication fails: unable to obtain a session (gssapi auth failed:
> requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='
> koji.fedoraproject.org', port=443): Read timed out. (read timeout=60))
>
> Anyone else seeing this?

Yes, up and down most of the morning.

Rich.

[Luna Jernberg]

Did notice it too, and reported it on Pagure and IRC and smooge said
it was sad due to the database using too much RAM memory

On Wed, Nov 16, 2022 at 2:20 PM Richard Shaw  wrote:
> I didn't see any other threads but I can't have been the first to notice...
>
> I can't access the koji web interface or initiate builds...
>
> Kerberos authentication fails: unable to obtain a session (gssapi auth failed: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='koji.fedoraproject.org', port=443): Read timed out. (read timeout=60))
>
> Anyone else seeing this?
>
> Thanks,
> Richard
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Stephen Smoogen]

On Wed, 16 Nov 2022 at 08:20, Richard Shaw  wrote:
> I didn't see any other threads but I can't have been the first to notice...
>
> I can't access the koji web interface or initiate builds...
>
> Kerberos authentication fails: unable to obtain a session (gssapi auth
> failed: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='
> koji.fedoraproject.org', port=443): Read timed out. (read timeout=60))
>
> Anyone else seeing this?

Currently the backend database is running out of memory (128GB) and CPU (90
cpus) and is running at a continuous load average of over 300. The system
earlier crashed hard and needed to be rebooted but that did not seem to
help any. The release engineers are looking into it but I do not have any
information than that.
> Thanks,
> Richard
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Kevin Fenzi]

Everything should be back up now...

I'm watching it for a bit before I declare things are back fully to
normal though. ;)

Do note that mailing list threads aren't a great place to report
outages. Please report them to the infrastructure ticket tracker: 
https://pagure.io/fedora-infrastructure/

Thanks.

kevin

[Richard Shaw]

On Wed, Nov 16, 2022 at 11:32 AM Kevin Fenzi  wrote:
> Everything should be back up now...
>
> I'm watching it for a bit before I declare things are back fully to
> normal though. ;)
>
> Do note that mailing list threads aren't a great place to report
> outages. Please report them to the infrastructure ticket tracker:
> https://pagure.io/fedora-infrastructure/

I recently switched internet providers so I was looking for an ACK that it
wasn't just me.

But on a related note, it can be somewhat difficult to remember all the
different sites and which is appropriate for what when you don't deal with
them on a regular basis..

Thanks,
Richard

[Kevin Fenzi]

On Wed, Nov 16, 2022 at 11:37:43AM -0600, Richard Shaw wrote:
> On Wed, Nov 16, 2022 at 11:32 AM Kevin Fenzi  wrote:
> > Everything should be back up now...
> >
> > I'm watching it for a bit before I declare things are back fully to
> > normal though. ;)
> >
> > Do note that mailing list threads aren't a great place to report
> > outages. Please report them to the infrastructure ticket tracker:
> > https://pagure.io/fedora-infrastructure/
>
> I recently switched internet providers so I was looking for an ACK that it
> wasn't just me.
>
> But on a related note, it can be somewhat difficult to remember all the
> different sites and which is appropriate for what when you don't deal with
> them on a regular basis..

Yeah.

I'm not sure how to make things more visible, but really the best link
to know is the one you get to via docs.fedoraproject.org, engineering
teams, fedora infra, working with fedora infra. ie,
https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/

kevin

[Gary Buhrmaster]

On Wed, Nov 16, 2022 at 5:38 PM Richard Shaw  wrote:
> I recently switched internet providers so I was looking for an ACK that it wasn't just me.

I am curious, did you try https://status.fedoraproject.org and did
it show the outage at the time (since it is manually updated, it
would typically lag the actual status, but it might have provided
a hint)?