What's up with koschei?

[Fabio Valentini]

Hi everybody,

I've noticed that koschei doesn't seem to have scheduled any scratch
builds since around the fedora 32 mass rebuild, and it also hasn't
picked up the f32 branch yet.

Is there a known "outage" or is it just sitting there idle for some
reason? I really like to know when my packages start to FTBFS ...

Fabio

[Mikolaj Izdebski]

On Mon, Feb 17, 2020 at 10:27 AM Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  I've noticed that koschei doesn't seem to have scheduled any
> scratch
>  builds since around the fedora 32 mass rebuild, and it also hasn't
>  picked up the f32 branch yet. 
Koschei collections are manipulated manually by Koschei admins
(members of sysadmin-koschei FAS group). Since all sysadmin-koschei
members were on vacation during and after Fedora 32 branching,
corresponding collections have not been created yet. I expect to
create them this week.

>  Is there a known "outage" or is it just sitting there idle
> for some
>  reason? I really like to know when my packages start to FTBFS ... 
That was unplanned outage - someone scaled scheduler service down to
zero replicas, so it was not running. Unfortunately after move to
OpenShift we don't have decent monitoring for Koschei, so we don't
have a way of seeing such problems until they are reported by users.
I've just scaled scheduler up and it seems to be running again.

Thanks for the report.

--
Mikolaj Izdebski

[Fabio Valentini]

On Mon, Feb 17, 2020 at 1:12 PM Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:
>  On Mon, Feb 17, 2020 at 10:27 AM Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  > I've noticed that koschei doesn't seem to have scheduled any scratch
>  > builds since around the fedora 32 mass rebuild, and it also hasn't
>  > picked up the f32 branch yet. 
(snip)

>  Koschei collections are manipulated manually by Koschei admins
>  (members of sysadmin-koschei FAS group). Since all sysadmin-koschei
>  members were on vacation during and after Fedora 32 branching,
>  corresponding collections have not been created yet. I expect to
>  create them this week. 
I didn't know that adding new releases is a manual process. Thanks for the info.

>  > Is there a known "outage" or is it just sitting there
> idle for some
>  > reason? I really like to know when my packages start to FTBFS ...
>
>  That was unplanned outage - someone scaled scheduler service down to
>  zero replicas, so it was not running. Unfortunately after move to
>  OpenShift we don't have decent monitoring for Koschei, so we don't
>  have a way of seeing such problems until they are reported by users.
>  I've just scaled scheduler up and it seems to be running again. 
Awesome, thanks for fixing it!

Fabio

> Thanks for the report.
>
> --
> Mikolaj Izdebski
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Kevin Fenzi]

On Mon, Feb 17, 2020 at 01:09:00PM +0100, Mikolaj Izdebski wrote:
>  On Mon, Feb 17, 2020 at 10:27 AM Fabio Valentini
> <decathorpe(a)gmail.com&gt; wrote:
>  > I've noticed that koschei doesn't seem to have scheduled any scratch
>  > builds since around the fedora 32 mass rebuild, and it also hasn't
>  > picked up the f32 branch yet.
>  
>  Koschei collections are manipulated manually by Koschei admins
>  (members of sysadmin-koschei FAS group). Since all sysadmin-koschei
>  members were on vacation during and after Fedora 32 branching,
>  corresponding collections have not been created yet. I expect to
>  create them this week.
>  
>  > Is there a known "outage" or is it just sitting there idle for some
>  > reason? I really like to know when my packages start to FTBFS ...
>  
>  That was unplanned outage - someone scaled scheduler service down to
>  zero replicas, so it was not running. Unfortunately after move to
>  OpenShift we don't have decent monitoring for Koschei, so we don't
>  have a way of seeing such problems until they are reported by users.
>  I've just scaled scheduler up and it seems to be running again. 
We scaled it down because it was causing builders to have full disks and
was breaking things for the mass rebuild. 

https://github.com/fedora-infra/koschei/issues/310

I fear if nothing has changed we will start getting full disks on builds
again. ;( 

kevin