Greetings.
I just thought I would share a short status update on koji builders with everyone. As many of you know, we like to keep the Fedora koji builders on the current most recent stable Fedora release. This is to make sure we have compatible rpm, etc to build rawhide and all our stable updates.
TLDR summary: All Fedora koji builders are now on Fedora 31 and are using only python3 (the hub is still python2 however).
Due to a bug in the linux kernel on armv7, we never upgraded to Fedora 30 when it was released, so builders stayed on Fedora 29 for the last cycle. Happily that bug was tracked down and fixed and I have been moving them to Fedora 31 for the past month or so.
Normally it doesn't take to long to reinstall everything (Thanks ansible!) but this time it's been a longer road due to: New aarch64 hardware that needed racking/cabling/setup (many thanks to Smooge for getting this done), firmware updates for most all hardware, RHEL8 reinstalls for virthosts that run buildvm's, and many other items.
We still have pending:
* Some more aarch64 hardware to bring online early in the new year. * A bug around armv7 buildvm's with more than 24GB memory. While thats being investigated all the buildvm-armv7 instances have just 24GB memory. * A bug around power9 hosts, causing our cloud/container image builds to fail. This was thought fixed, but doesn't seem to be yet. * Daily db dumps still cause slowness, likely in the new year we will schedule an outage and patition the problem tables.
A few stats:
155 total enabled builders 47 x86_64 builders 32 ppc64le builders 32 aarch64 builders 27 armhfp builders 24 s390x builders
Around 10,000 builds a month. Close to 1.1 million builds since fc6 days
Happy building.
On Fri, Dec 13, 2019 at 7:22 PM Kevin Fenzi kevin@scrye.com wrote:
Greetings.
Hi Kevin,
I just thought I would share a short status update on koji builders with everyone. As many of you know, we like to keep the Fedora koji builders on the current most recent stable Fedora release. This is to make sure we have compatible rpm, etc to build rawhide and all our stable updates.
TLDR summary: All Fedora koji builders are now on Fedora 31 and are using only python3 (the hub is still python2 however).
That's great news, thanks for the update, and for keeping fedora infra running smoothly! :)
Due to a bug in the linux kernel on armv7, we never upgraded to Fedora 30 when it was released, so builders stayed on Fedora 29 for the last cycle. Happily that bug was tracked down and fixed and I have been moving them to Fedora 31 for the past month or so.
Normally it doesn't take to long to reinstall everything (Thanks ansible!) but this time it's been a longer road due to: New aarch64 hardware that needed racking/cabling/setup (many thanks to Smooge for getting this done), firmware updates for most all hardware, RHEL8 reinstalls for virthosts that run buildvm's, and many other items.
We still have pending:
- Some more aarch64 hardware to bring online early in the new year.
- A bug around armv7 buildvm's with more than 24GB memory. While thats
being investigated all the buildvm-armv7 instances have just 24GB memory.
- A bug around power9 hosts, causing our cloud/container image builds to
fail. This was thought fixed, but doesn't seem to be yet.
- Daily db dumps still cause slowness, likely in the new year we will
schedule an outage and patition the problem tables.
A few stats:
155 total enabled builders 47 x86_64 builders 32 ppc64le builders 32 aarch64 builders 27 armhfp builders 24 s390x builders
Side note: Considering that there are comparatively few armv7hl builders, I wonder why a rather big share of my noarch builds gets scheduled to run on them. Naively, I'd think that armv7hl should be busy doing builds for "archful" stuff since they're the slowest builders and there are the fewest of them, except for s390x. Since most of the noarch stuff I do for fedora nowadays is Java (*sigh*), these Java-on-arm builds are rather slow. Maybe that's bad luck, or maybe the task scheduler hates me :)
Fabio
Around 10,000 builds a month. Close to 1.1 million builds since fc6 days
Happy building. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Sun, Dec 15, 2019 at 10:24:16AM +0100, Fabio Valentini wrote:
Side note: Considering that there are comparatively few armv7hl builders, I wonder why a rather big share of my noarch builds gets scheduled to run on them. Naively, I'd think that armv7hl should be busy doing builds for "archful" stuff since they're the slowest
I'm not sure why that is. I would expect them to be busy more as well. Do note that we are bringing some more hardware on line after the new year, which should give us more armv7 builders.
builders and there are the fewest of them, except for s390x.
s390x now is a great deal faster. It's often the one that finishes right after x86_64/i686.
Since most of the noarch stuff I do for fedora nowadays is Java (*sigh*), these Java-on-arm builds are rather slow. Maybe that's bad luck, or maybe the task scheduler hates me :)
Yeah, not sure. We can try and investigate more.
kevin
On Thu, 19 Dec 2019 12:10:00 -0800 Kevin Fenzi kevin@scrye.com wrote:
On Sun, Dec 15, 2019 at 10:24:16AM +0100, Fabio Valentini wrote:
Side note: Considering that there are comparatively few armv7hl builders, I wonder why a rather big share of my noarch builds gets scheduled to run on them. Naively, I'd think that armv7hl should be busy doing builds for "archful" stuff since they're the slowest
I'm not sure why that is. I would expect them to be busy more as well. Do note that we are bringing some more hardware on line after the new year, which should give us more armv7 builders.
builders and there are the fewest of them, except for s390x.
s390x now is a great deal faster. It's often the one that finishes right after x86_64/i686.
even ahead x86_64/i686 sometimes :-)
Dan
On Thu, 19 Dec 2019 at 21:13, Dan Horák dan@danny.cz wrote:
On Thu, 19 Dec 2019 12:10:00 -0800 Kevin Fenzi kevin@scrye.com wrote:
On Sun, Dec 15, 2019 at 10:24:16AM +0100, Fabio Valentini wrote:
Side note: Considering that there are comparatively few armv7hl builders, I wonder why a rather big share of my noarch builds gets scheduled to run on them. Naively, I'd think that armv7hl should be busy doing builds for "archful" stuff since they're the slowest
I'm not sure why that is. I would expect them to be busy more as well. Do note that we are bringing some more hardware on line after the new year, which should give us more armv7 builders.
builders and there are the fewest of them, except for s390x.
s390x now is a great deal faster. It's often the one that finishes right after x86_64/i686.
even ahead x86_64/i686 sometimes :-)
This is great to hear! Until now s390x was the slowest arch for Java by quite a wide margin; with the special Java 8 JIT package[1] installed, even 32bit arm was quicker than s390x :-o
[1] "java-1.8.0-openjdk-aarch32" -- https://koji.fedoraproject.org/koji/packageinfo?packageID=22627
-- Mat Booth http://fedoraproject.org/get-fedora
On Fri, 20 Dec 2019 01:39:42 +0000 Mat Booth fedora@matbooth.co.uk wrote:
On Thu, 19 Dec 2019 at 21:13, Dan Horák dan@danny.cz wrote:
On Thu, 19 Dec 2019 12:10:00 -0800 Kevin Fenzi kevin@scrye.com wrote:
On Sun, Dec 15, 2019 at 10:24:16AM +0100, Fabio Valentini wrote:
Side note: Considering that there are comparatively few armv7hl builders, I wonder why a rather big share of my noarch builds gets scheduled to run on them. Naively, I'd think that armv7hl should be busy doing builds for "archful" stuff since they're the slowest
I'm not sure why that is. I would expect them to be busy more as well. Do note that we are bringing some more hardware on line after the new year, which should give us more armv7 builders.
builders and there are the fewest of them, except for s390x.
s390x now is a great deal faster. It's often the one that finishes right after x86_64/i686.
even ahead x86_64/i686 sometimes :-)
This is great to hear! Until now s390x was the slowest arch for Java by quite a wide margin; with the special Java 8 JIT package[1] installed, even 32bit arm was quicker than s390x :-o
because there is no JIT for s390x in OpenJDK 1.8.0, it has been added in some later version (OpenJDK 10?). So until this version is the new default Java, s390x will be slow for Java.
Dan
On 19. 12. 19 21:10, Kevin Fenzi wrote:
On Sun, Dec 15, 2019 at 10:24:16AM +0100, Fabio Valentini wrote:
Side note: Considering that there are comparatively few armv7hl builders, I wonder why a rather big share of my noarch builds gets scheduled to run on them. Naively, I'd think that armv7hl should be busy doing builds for "archful" stuff since they're the slowest
I'm not sure why that is. I would expect them to be busy more as well.
I have the same experience with Python noarch builds.
I have figured out that it is quite deterministic:
When I need the build quickly and/or there are hundreds of slow-ish tests, I get arm. When I don't need the build quickly and/or there are no tests at all and it builds in seconds anywhere, I get x86.
Based on that, I've determined that it is just a bias :D
-
Dan Horák -
Fabio Valentini -
Kevin Fenzi -
Mat Booth -
Miro Hrončok