2:53 a.m.
Hello folks,
during our Nest FESCo session, we've talked about enabling Koschei [1] for all
packages automatically.
There seem to be a consensus by FESCo members, that it would be a good thing.
What would it take?
1) Koji resources
I think we can try to enable this and see if it burns. I think ti won't.
2) One-time enablement of all existing packages
That should be doable. Right?
3) Automatic enablement of all new packages
That should be just a matter of changing the defaults. Correct?
Can we do this? How can I help?
[1] https://fedoraproject.org/wiki/Koschei
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
8:58 a.m.
On Thu, Aug 25, 2022 at 9:58 AM Miro Hrončok <mhroncok(a)redhat.com> wrote:
Hello folks,
during our Nest FESCo session, we've talked about enabling Koschei [1] for all
packages automatically.
There seem to be a consensus by FESCo members, that it would be a good thing.
What would it take?
1) Koji resources
I think we can try to enable this and see if it burns. I think ti won't.
Doesn't koschei have some built-in rate-limiting?
If necessary, we could probably just adjust the priority threshold
above which scratch builds are triggered.
2) One-time enablement of all existing packages
That should be doable. Right?
Yeah that should be pretty simple. Get list of source packages in
$release, paste it into the "Add packages" page, set release combobox
to $release, submit.
I do that regularly for all Rust packages.
3) Automatic enablement of all new packages
That should be just a matter of changing the defaults. Correct?
I think so. koschei already knows about all packages - I think it
syncs with PDC. So that process could possibly be adapted to not only
add the package to the koschei database, but also set the "tracked"
boolean to "true" by default instead of "false" by default.
Can we do this? How can I help?
I think we can, and I think we should.
Even though scratch builds only run on x86_64, ppc64le, and aarch64
(and don't expose s390x and i686 build failures), it's still a very
valuable service.
I use it extensively to monitor the Rust stack for new build failures
(because I'm pushing dozens of updates every week and I want to know
about unintended breakage ASAP).
If there's something that I can do to help make "koschei tracking for
all packages on all branches by default" happen, I'll help too :)
Fabio
9:34 a.m.
On Thu, Aug 25, 2022 15:58:12 +0200, Fabio Valentini wrote:
Yeah that should be pretty simple. Get list of source packages in
$release, paste it into the "Add packages" page, set release combobox
to $release, submit.
I do that regularly for all Rust packages.
The rust-sig packages are also set to automatically be tracked by
Koschei from the looks of it and I expect this can then be done for all
packages instead of for individual SIGs:
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/k...
--
Thanks,
Regards,
Ankur Sinha "FranciscoD" (He / Him / His) |
https://fedoraproject.org/wiki/User:Ankursinha
Time zone: Europe/London
9:38 a.m.
On Thu, Aug 25, 2022 at 4:34 PM Ankur Sinha <sanjay.ankur(a)gmail.com> wrote:
On Thu, Aug 25, 2022 15:58:12 +0200, Fabio Valentini wrote:
> Yeah that should be pretty simple. Get list of source packages in
> $release, paste it into the "Add packages" page, set release combobox
> to $release, submit.
> I do that regularly for all Rust packages.
The rust-sig packages are also set to automatically be tracked by
Koschei from the looks of it and I expect this can then be done for all
packages instead of for individual SIGs:
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/k...
As far as I understand, this only automatically adds packages that are
associated with the "@rust-sig" group on dist-git to the "rust-sig"
group on koschei.
We still need to manually enable tracking for new packages, but
syncing group membership is automatic.
Fabio
3:09 a.m.
Dne 25. 08. 22 v 16:38 Fabio Valentini napsal(a):
On Thu, Aug 25, 2022 at 4:34 PM Ankur Sinha
<sanjay.ankur(a)gmail.com> wrote:
> On Thu, Aug 25, 2022 15:58:12 +0200, Fabio Valentini wrote:
>> Yeah that should be pretty simple. Get list of source packages in
>> $release, paste it into the "Add packages" page, set release combobox
>> to $release, submit.
>> I do that regularly for all Rust packages.
> The rust-sig packages are also set to automatically be tracked by
> Koschei from the looks of it and I expect this can then be done for all
> packages instead of for individual SIGs:
>
>
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/k...
As far as I understand, this only automatically adds packages that are
associated with the "@rust-sig" group on dist-git to the
"rust-sig"
group on koschei.
We still need to manually enable tracking for new packages, but
syncing group membership is automatic.
https://pagure.io/fedora-infra/ansible/blob/c2c063d22fe9a36b167fd27496aa1...
https://github.com/fedora-infra/koschei/blob/master/bin/koschei-track-group
Checking these two ^^ I'd say that the whole group is automatically tracked.
Vít
Fabio
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
11:36 p.m.
On Fri, Aug 26, 2022 at 10:09 AM Vít Ondruch <vondruch(a)redhat.com> wrote:
Dne 25. 08. 22 v 16:38 Fabio Valentini napsal(a):
> On Thu, Aug 25, 2022 at 4:34 PM Ankur Sinha <sanjay.ankur(a)gmail.com> wrote:
>> On Thu, Aug 25, 2022 15:58:12 +0200, Fabio Valentini wrote:
>>> Yeah that should be pretty simple. Get list of source packages in
>>> $release, paste it into the "Add packages" page, set release
combobox
>>> to $release, submit.
>>> I do that regularly for all Rust packages.
>> The rust-sig packages are also set to automatically be tracked by
>> Koschei from the looks of it and I expect this can then be done for all
>> packages instead of for individual SIGs:
>>
>>
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/k...
> As far as I understand, this only automatically adds packages that are
> associated with the "@rust-sig" group on dist-git to the
"rust-sig"
> group on koschei.
> We still need to manually enable tracking for new packages, but
> syncing group membership is automatic.
https://pagure.io/fedora-infra/ansible/blob/c2c063d22fe9a36b167fd27496aa1...
https://github.com/fedora-infra/koschei/blob/master/bin/koschei-track-group
Checking these two ^^ I'd say that the whole group is automatically tracked.
That is correct.
All packages for which rust-sig dist-git group is a watcher
(configurable in dist-git) are added to rust-sig group in Koschei.
Then all packages that belong to the rust-sig group in Koschei are set
as tracked.
This is done by cron jobs scheduled every 3 hours.
Anyone can request packages meeting certain criteria (eg. name
matching some pattern, or owned/watched by a particular person/group)
to be auto-tracked by Koschei or added to a particular group of
packages.
--
Mikolaj Izdebski
Vít
>
> Fabio
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
10:54 a.m.
On Thu, 2022-08-25 at 09:53 +0200, Miro Hrončok wrote:
Hello folks,
during our Nest FESCo session, we've talked about enabling Koschei [1] for all
packages automatically.
There seem to be a consensus by FESCo members, that it would be a good thing.
What would it take?
1) Koji resources
I think we can try to enable this and see if it burns. I think ti won't.
2) One-time enablement of all existing packages
That should be doable. Right?
3) Automatic enablement of all new packages
That should be just a matter of changing the defaults. Correct?
Can we do this? How can I help?
Big +1. Even for 1), if it turns out things burn, we now have a great
case to ask for more resources - we're not just theoretically trying to
bump things up against possible capacity requirements, we need to fix
something that's a problem right now! ;)
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
10:21 p.m.
On Thu, Aug 25, 2022 at 7:57 AM Miro Hrončok <mhroncok(a)redhat.com> wrote:
Hello folks,
during our Nest FESCo session, we've talked about enabling Koschei [1] for all
packages automatically.
I am not anywhere near the deciders group, but this
(enabling Koschei by default) just makes too much
sense not to do for the vast majority of cases (I have
no doubt there are edge cases (there are always
edge cases) where disabling Koschei is the right
choice, too).
And while I am a small time packager, I enable Koschei
for all my packages (CI is not a panacea, of course,
no matter what some advocates might try to assert).
8:22 a.m.
On Thu, 25 Aug 2022 at 03:58, Miro Hrončok <mhroncok(a)redhat.com> wrote:
Hello folks,
during our Nest FESCo session, we've talked about enabling Koschei [1] for
all
packages automatically.
There seem to be a consensus by FESCo members, that it would be a good
thing.
What would it take?
1) Koji resources
I think we can try to enable this and see if it burns. I think ti won't.
It will, but we will deal with it. Koschei builds have had to be throttled
down multiple times in the past due to the fact that all those builds
working or failed take up some disk space on builders for a bit. Get a lot
of churn and koschei builds and you end up with builders without disk space
and various developers who are saying yes! in the past yelling about not
being able to do their real work :). We will just have to figure out how to
throttle things down safely to make it work. The current
koji+all-other-stuff is a batch operating system (line up the tasks and do
them until they are done) which we treat like a time-share which only holds
when things are 'low' usage.
Long term I think we are going to need to figure out ways to get the fedora
buildsystem safely and securely into various clouds to allow for many more
builders and resources. [This isn't just koji, but the assumptions that all
the other tools in the fedora system from the 'dashboard' to the
koji+bodhi+osbs+odcs+pungi+etc+etc build system that all the resources are
close and local in a 'sealed' firewalled off space. ]
--
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren
9:09 a.m.
Dne 26. 08. 22 v 15:22 Stephen Smoogen napsal(a):
On Thu, 25 Aug 2022 at 03:58, Miro Hrončok <mhroncok(a)redhat.com> wrote:
Hello folks,
during our Nest FESCo session, we've talked about enabling Koschei
[1] for all
packages automatically.
There seem to be a consensus by FESCo members, that it would be a
good thing.
What would it take?
1) Koji resources
I think we can try to enable this and see if it burns. I think ti
won't.
It will, but we will deal with it. Koschei builds have had to be
throttled down multiple times in the past due to the fact that all
those builds working or failed take up some disk space on builders for
a bit.
Unfortunately this took toll on Koschei usability, because the logs are
not preserved :/ I wish the logs were extracted from Koji and stored
somewhere where they can be kept for more then two weeks.
Just yesterday, I was looking into rubygem-rubyzip failures [1], which
happened every ~8th build. Without logs, Koschei was not really helpful.
Vít
P.S. for the curious, while the package is noarch, the build failed when
executed on s390x. IOW LE / BE issue.
[1] https://koschei.fedoraproject.org/package/rubygem-rubyzip
2:21 p.m.
On 8/26/22 09:22, Stephen Smoogen wrote:
On Thu, 25 Aug 2022 at 03:58, Miro Hrončok
<mhroncok(a)redhat.com> wrote:
> Hello folks,
>
> during our Nest FESCo session, we've talked about enabling Koschei [1] for
> all
> packages automatically.
>
> There seem to be a consensus by FESCo members, that it would be a good
> thing.
>
> What would it take?
>
> 1) Koji resources
>
> I think we can try to enable this and see if it burns. I think ti won't.
>
>
It will, but we will deal with it. Koschei builds have had to be throttled
down multiple times in the past due to the fact that all those builds
working or failed take up some disk space on builders for a bit. Get a lot
of churn and koschei builds and you end up with builders without disk space
and various developers who are saying yes! in the past yelling about not
being able to do their real work :). We will just have to figure out how to
throttle things down safely to make it work. The current
koji+all-other-stuff is a batch operating system (line up the tasks and do
them until they are done) which we treat like a time-share which only holds
when things are 'low' usage.
Long term I think we are going to need to figure out ways to get the fedora
buildsystem safely and securely into various clouds to allow for many more
builders and resources. [This isn't just koji, but the assumptions that all
the other tools in the fedora system from the 'dashboard' to the
koji+bodhi+osbs+odcs+pungi+etc+etc build system that all the resources are
close and local in a 'sealed' firewalled off space. ]
How bursty is the builder workload? Also, if you go this route, I strongly
recommend enforcing CA pinning (not public key pinning!) to the appropriate
Google/Amazon roots.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
1:15 p.m.
On Thu, Aug 25, 2022 at 9:53 AM Miro Hrončok <mhroncok(a)redhat.com> wrote:
Hello folks,
during our Nest FESCo session, we've talked about enabling Koschei [1] for all
packages automatically.
There seem to be a consensus by FESCo members, that it would be a good thing.
What would it take?
1) Koji resources
I think we can try to enable this and see if it burns. I think ti won't.
I don't expect Koschei would use much more Koji resources than it already does.
Koschei maintains a priority queue of package rebuilds and submits
scratch builds as available Koji resources allow.
The queue is almost never empty - there are almost always some
packages to rebuild.
An increased number of tracked packages would simply grow the queue,
but would not lead to submitting much more scratch builds.
IOW, packages would simply be rebuilt less often.
To submit more scratch builds we would need larger builder capacity.
This doesn't necessarily mean more or better hardware.
Better Koji configuration would help a lot.
We have some very powerful builders with up to 224 processors, but
their capacity is set to 2.
This means that the builder stops accepting new tasks once load gets
to 2, which is less than 1 %.
Example buildhw-a64-20.iad2.fedoraproject.org
Capacity is 2, check with: koji hostinfo buildhw-a64-20.iad2.fedoraproject.org
224 CPUs, load average: 2.04, 2.07, 2.05
memory: 251Gi total, 7.3Gi used, 242Gi available
Yet, at the time of writing the builder is marked as not ready (!!)
for taking more builds due to exceeded capacity.
This is not an individual case, we have many builders like that.
Making all packages as tracked without increasing Koji builder
capacity will limit Koschei
usefulness for packagers that care about Koschei enough to manually opt-in.
2) One-time enablement of all existing packages
That should be doable. Right?
3) Automatic enablement of all new packages
That should be just a matter of changing the defaults. Correct?
I can implement points 2 and 3 easily, as long as there is consensus to do so.
--
Mikolaj Izdebski
Can we do this? How can I help?
[1] https://fedoraproject.org/wiki/Koschei
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
8:56 a.m.
On Tue, 30 Aug 2022 at 14:16, Mikolaj Izdebski <mizdebsk(a)redhat.com> wrote:
On Thu, Aug 25, 2022 at 9:53 AM Miro Hrončok
<mhroncok(a)redhat.com> wrote:
To submit more scratch builds we would need larger builder capacity.
This doesn't necessarily mean more or better hardware.
Better Koji configuration would help a lot.
We have some very powerful builders with up to 224 processors, but
their capacity is set to 2.
This means that the builder stops accepting new tasks once load gets
to 2, which is less than 1 %.
Example buildhw-a64-20.iad2.fedoraproject.org
Capacity is 2, check with: koji hostinfo
buildhw-a64-20.iad2.fedoraproject.org
224 CPUs, load average: 2.04, 2.07, 2.05
memory: 251Gi total, 7.3Gi used, 242Gi available
Yet, at the time of writing the builder is marked as not ready (!!)
for taking more builds due to exceeded capacity.
This is not an individual case, we have many builders like that.
I believe there are 2 builders with 224 CPU's. They are both
'prototype'
aarch64 systems we got on loan from a vendor and are tempermental. They do
not have good disk IO and do not have the capability of increasing the disk
IO. This means that you have a very very fast cpu and slow as treacle disk
io. The network on them has also been 'fun' where if the net access got too
high, it would require a complete power cycle with the plugs pulled out
because the BMC is not licensed.
Most of the 'big' builders I remember are similar in hardware.. They look
great on paper, but if you try to give them a large capacity you end up
with even slower builds because they are loaners or spares which are good
for specific workloads and not general usage.
--
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren
2:46 p.m.
On Wed, Aug 31, 2022 at 3:57 PM Stephen Smoogen <ssmoogen(a)redhat.com> wrote:
On Tue, 30 Aug 2022 at 14:16, Mikolaj Izdebski <mizdebsk(a)redhat.com> wrote:
>
> On Thu, Aug 25, 2022 at 9:53 AM Miro Hrončok <mhroncok(a)redhat.com> wrote:
>
> To submit more scratch builds we would need larger builder capacity.
> This doesn't necessarily mean more or better hardware.
> Better Koji configuration would help a lot.
> We have some very powerful builders with up to 224 processors, but
> their capacity is set to 2.
> This means that the builder stops accepting new tasks once load gets
> to 2, which is less than 1 %.
>
> Example buildhw-a64-20.iad2.fedoraproject.org
> Capacity is 2, check with: koji hostinfo buildhw-a64-20.iad2.fedoraproject.org
> 224 CPUs, load average: 2.04, 2.07, 2.05
> memory: 251Gi total, 7.3Gi used, 242Gi available
> Yet, at the time of writing the builder is marked as not ready (!!)
> for taking more builds due to exceeded capacity.
>
> This is not an individual case, we have many builders like that.
>
I believe there are 2 builders with 224 CPU's. They are both 'prototype'
aarch64 systems we got on loan from a vendor and are tempermental. They do not have good
disk IO and do not have the capability of increasing the disk IO. This means that you have
a very very fast cpu and slow as treacle disk io. The network on them has also been
'fun' where if the net access got too high, it would require a complete power
cycle with the plugs pulled out because the BMC is not licensed.
Slow disk could be worked around by using tmpfs for mock chroots -
there is lots of free memory.
I was not aware of the networking issues.
--
Mikolaj Izdebski
Most of the 'big' builders I remember are similar in hardware.. They look great
on paper, but if you try to give them a large capacity you end up with even slower builds
because they are loaners or spares which are good for specific workloads and not general
usage.
--
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle. -- Ian
MacClaren
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
9:02 a.m.
On 30. 08. 22 20:15, Mikolaj Izdebski wrote:
> 2) One-time enablement of all existing packages
>
> That should be doable. Right?
>
>
> 3) Automatic enablement of all new packages
>
> That should be just a matter of changing the defaults. Correct?
I can implement points 2 and 3 easily, as long as there is consensus to do so.
I plan to propose a Fedora Linux 38 change proposal for this to get consensus.
Should I list you as a change owner?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
2:36 p.m.
On Wed, Aug 31, 2022 at 4:03 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
On 30. 08. 22 20:15, Mikolaj Izdebski wrote:
>> 2) One-time enablement of all existing packages
>>
>> That should be doable. Right?
>>
>>
>> 3) Automatic enablement of all new packages
>>
>> That should be just a matter of changing the defaults. Correct?
> I can implement points 2 and 3 easily, as long as there is consensus to do so.
I plan to propose a Fedora Linux 38 change proposal for this to get consensus.
Should I list you as a change owner?
I don't think the Change process is needed for this infrastructure change.
But if you like to, feel free to. I can be a co-owner if you like.
My plan was to bring this topic to the infrastructure list and gather
more feedback there.
Unless there were any objections I would proceed to implement the
proposal after Fedora 37 beta freeze - end of September.
I would rather not implement this during infra freeze as this change
is likely to put more load on services Koschei depends on.
--
Mikolaj Izdebski
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
602
days inactive
609
days old
17 comments
11 participants
participants (11)
-
Adam Williamson -
Ankur Sinha -
Demi Marie Obenour -
Fabio Valentini -
Gary Buhrmaster -
Michael Catanzaro -
Mikolaj Izdebski -
Miro Hrončok -
Stephen Smoogen -
Vitaly Zaitsev -
Vít Ondruch