10:07 a.m.
Hi All,
We are planning to change the way Alternate Architectures (non x86_64)
are handled
in terms of "primary" vs "secondary". The definition of what is
primary or secondary
is already handled more in terms of the build artifact outputs (images, LiveCDs,
installers, containers etc) for i686 deliverables. As part of this redefinition
this means that the location in "koji instances" of the rpm builds is removed
as
a part of the definition requirement of what constitutes
primary/secondary and the
architectures are named "Alternate Architectures" (and Experimental
architectures
for the likes of MIPs/RISC-V) as opposed to primary/secondary. As a
result of this
change it is planned to merge the old "secondary" koji instances into a single
koji instance along with all the current "Primary" architectures.
All the details of the proposal along with FAQ have been put on a wiki
page here[1]
so please go and read it and ask any questions that aren't answered in
the FAQ here.
Regards,
Peter
[1] https://fedoraproject.org/wiki/Architectures/RedefiningSecondaryArchitect...
10:27 a.m.
On 4 August 2016 at 11:07, Peter Robinson <pbrobinson(a)gmail.com> wrote:
All the details of the proposal along with FAQ have been put on a
wiki
page here[1]
so please go and read it and ask any questions that aren't answered in
the FAQ here.
Regards,
Peter
[1] https://fedoraproject.org/wiki/Architectures/RedefiningSecondaryArchitect...
--
What is the update for this statement:
Q: When will the new ARMv7 builders be in place?
A: Soon! The current plan is mid to late July. This proposal isn't
impacted by this as ARMv7 is already a primary architecture.
since we are past July.. is it July 2017 :)?
devel mailing list
devel(a)lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
--
Stephen J Smoogen.
12:27 p.m.
On Thu, Aug 4, 2016 at 10:27 AM, Stephen John Smoogen <smooge(a)gmail.com> wrote:
On 4 August 2016 at 11:07, Peter Robinson
<pbrobinson(a)gmail.com> wrote:
> All the details of the proposal along with FAQ have been put on a wiki
> page here[1]
> so please go and read it and ask any questions that aren't answered in
> the FAQ here.
>
> Regards,
> Peter
>
> [1] https://fedoraproject.org/wiki/Architectures/RedefiningSecondaryArchitect...
> --
What is the update for this statement:
Q: When will the new ARMv7 builders be in place?
A: Soon! The current plan is mid to late July. This proposal isn't
impacted by this as ARMv7 is already a primary architecture.
since we are past July.. is it July 2017 :)?
That wiki was only setup July 21st (Mid to Late July), so probably go
with "Soon!" instead of any approximate date.
Setting up a number of m400 aarch64 nodes to provide ARMv7 virtual
builders is pretty cool.
So I certainly appreciate infra taking their time to get it right. :-)
--
-Jon Disnard
1:41 p.m.
On Thu, 2016-08-04 at 16:07 +0100, Peter Robinson wrote:
Hi All,
We are planning to change the way Alternate Architectures (non x86_64)
are handled
in terms of "primary" vs "secondary". The definition of what is
primary or secondary
is already handled more in terms of the build artifact outputs (images, LiveCDs,
installers, containers etc) for i686 deliverables. As part of this redefinition
this means that the location in "koji instances" of the rpm builds is removed
as
a part of the definition requirement of what constitutes
primary/secondary and the
architectures are named "Alternate Architectures" (and Experimental
architectures
for the likes of MIPs/RISC-V) as opposed to primary/secondary. As a
result of this
change it is planned to merge the old "secondary" koji instances into a single
koji instance along with all the current "Primary" architectures.
All the details of the proposal along with FAQ have been put on a wiki
page here[1]
so please go and read it and ask any questions that aren't answered in
the FAQ here.
I do have serious concerns about the impact of this in terms of build
failures. Reading the reply to " Q: Why do I have to worry about
s390x/powerpc/aarch64 when I didn't before?", it implies there will be
no change to koji in terms of build failures: i.e. a failure on *any*
arch will cause the entire build to be failed.
The answer...honestly does not convince me. I think the result will be
a combination both of an increase in failed builds and the issues
caused by them, and an increase in the number of packages which simply
disable building on an arch entirely due to a lack of will to deal with
build issues (and/or slow build time) on that arch.
With secondary koji instances, neither of these are major issues, and
secondary arch teams are able to work on build fixes for those arches
without the maintainers being bothered by them.
Has any consideration been given to the possibility of increasing
Koji's flexibility here, by allowing for arches to be designated as
non-fatal, so a package build failure on that arch would not cause the
task to be considered a failure?
1:49 p.m.
On Thu, 2016-08-04 at 11:41 -0700, Adam Williamson wrote:
On Thu, 2016-08-04 at 16:07 +0100, Peter Robinson wrote:
>
> Hi All,
>
> We are planning to change the way Alternate Architectures (non x86_64)
> are handled
> in terms of "primary" vs "secondary". The definition of what is
> primary or secondary
> is already handled more in terms of the build artifact outputs (images, LiveCDs,
> installers, containers etc) for i686 deliverables. As part of this redefinition
> this means that the location in "koji instances" of the rpm builds is
removed as
> a part of the definition requirement of what constitutes
> primary/secondary and the
> architectures are named "Alternate Architectures" (and Experimental
> architectures
> for the likes of MIPs/RISC-V) as opposed to primary/secondary. As a
> result of this
> change it is planned to merge the old "secondary" koji instances into a
single
> koji instance along with all the current "Primary" architectures.
>
> All the details of the proposal along with FAQ have been put on a wiki
> page here[1]
> so please go and read it and ask any questions that aren't answered in
> the FAQ here.
I do have serious concerns about the impact of this in terms of build
failures. Reading the reply to " Q: Why do I have to worry about
s390x/powerpc/aarch64 when I didn't before?", it implies there will be
no change to koji in terms of build failures: i.e. a failure on *any*
arch will cause the entire build to be failed.
Sorry, just saw there was a more specific entry for my concern,
"Q: Will a single arch failure affect the overall build failure?" Still
not 100% sure, but thanks for addressing it.
2:44 p.m.
On Thu, Aug 4, 2016 at 2:49 PM, Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
On Thu, 2016-08-04 at 11:41 -0700, Adam Williamson wrote:
> On Thu, 2016-08-04 at 16:07 +0100, Peter Robinson wrote:
> >
> > Hi All,
> >
> > We are planning to change the way Alternate Architectures (non x86_64)
> > are handled
> > in terms of "primary" vs "secondary". The definition of what
is
> > primary or secondary
> > is already handled more in terms of the build artifact outputs (images,
LiveCDs,
> > installers, containers etc) for i686 deliverables. As part of this redefinition
> > this means that the location in "koji instances" of the rpm builds is
removed as
> > a part of the definition requirement of what constitutes
> > primary/secondary and the
> > architectures are named "Alternate Architectures" (and Experimental
> > architectures
> > for the likes of MIPs/RISC-V) as opposed to primary/secondary. As a
> > result of this
> > change it is planned to merge the old "secondary" koji instances into
a single
> > koji instance along with all the current "Primary" architectures.
> >
> > All the details of the proposal along with FAQ have been put on a wiki
> > page here[1]
> > so please go and read it and ask any questions that aren't answered in
> > the FAQ here.
>
> I do have serious concerns about the impact of this in terms of build
> failures. Reading the reply to " Q: Why do I have to worry about
> s390x/powerpc/aarch64 when I didn't before?", it implies there will be
> no change to koji in terms of build failures: i.e. a failure on *any*
> arch will cause the entire build to be failed.
Sorry, just saw there was a more specific entry for my concern,
"Q: Will a single arch failure affect the overall build failure?" Still
not 100% sure, but thanks for addressing it.
I think the solid way to address this is to make each architecture
independent and don't stop the build for any arch if any other arch
fails. The total failure state can be figured out once all the arches
have completed and based on criteria on which ones are considered
fatal or not, it would make a judgement. This is how it is done in
Youri for Mageia. When we submit packages to build, all architectures
build. However, only a failure in i586 and x86_64 triggers the failed
state. Builds in armv5tl and armv7hl do not.
--
真実はいつも一つ！/ Always, there's only one truth!
2:48 p.m.
On Thu, 2016-08-04 at 15:44 -0400, Neal Gompa wrote:
I think the solid way to address this is to make each architecture
independent and don't stop the build for any arch if any other arch
fails. The total failure state can be figured out once all the arches
have completed and based on criteria on which ones are considered
fatal or not, it would make a judgement. This is how it is done in
Youri for Mageia. When we submit packages to build, all architectures
build. However, only a failure in i586 and x86_64 triggers the failed
state. Builds in armv5tl and armv7hl do not.
The page says that Koji will be modified to run all the per-arch build
tasks to completion even if one fails (as opposed to how it behaves
now, cancelling all the other arch tasks as soon as any one fails), but
a failure of any of them will still constitute a failure of the overall
task.
2:53 p.m.
On Thu, Aug 04, 2016 at 04:07:31PM +0100, Peter Robinson wrote:
I skimmed all this and I'm still a bit confused.
Will there be one Koji instance compiling for every (current primary +
current secondary) arch? Or will there be two instances, one for all
primary and one for all secondary?
Will a build failure on (say) aarch64 prevent my package from
progressing to Rawhide (x86_64), bodhi etc?
-*- -*- -*-
On the subject of alternate architectures, I'm making available Fedora
images available in virt-builder for aarch64, armv7l, ppc64 and
ppc64le. There is a complete set for Fedora 23, and a partial set for
Fedora 24 (booting problems on ppc64 - will be solved eventually).
You can run these up on x86_64 hosts quite easily. For an example of
how see:
https://rwmj.wordpress.com/2015/04/15/virt-builder-fedora-21-ppc64-and-pp...
(The virt-install method is now the recommended one. Don't run qemu
directly.)
-*- -*- -*-
On the subject of RISC-V, I'm still plugging away at this. It's
rather slow going, but you can take a look at:
http://git.annexia.org/?p=fedora-riscv.git;a=summary
There is nothing much usable at the moment, and many stumbling blocks.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
2:56 p.m.
On Thu, Aug 4, 2016 at 3:53 PM, Richard W.M. Jones <rjones(a)redhat.com> wrote:
On Thu, Aug 04, 2016 at 04:07:31PM +0100, Peter Robinson wrote:
> [1] https://fedoraproject.org/wiki/Architectures/RedefiningSecondaryArchitect...
I skimmed all this and I'm still a bit confused.
Will there be one Koji instance compiling for every (current primary +
current secondary) arch? Or will there be two instances, one for all
primary and one for all secondary?
From what I can tell, shadow Koji instances are going away entirely.
There will be one Koji system to rule them all.
Will a build failure on (say) aarch64 prevent my package from
progressing to Rawhide (x86_64), bodhi etc?
-*- -*- -*-
That's what Adam Williamson seems to indicate, which I expect to be
quite problematic.
--
真実はいつも一つ！/ Always, there's only one truth!
2:59 p.m.
On Thu, 2016-08-04 at 15:56 -0400, Neal Gompa wrote:
On Thu, Aug 4, 2016 at 3:53 PM, Richard W.M. Jones
<rjones(a)redhat.com> wrote:
>
> On Thu, Aug 04, 2016 at 04:07:31PM +0100, Peter Robinson wrote:
> >
> > [1]
https://fedoraproject.org/wiki/Architectures/RedefiningSecondaryArchitect...
>
> I skimmed all this and I'm still a bit confused.
>
> Will there be one Koji instance compiling for every (current primary +
> current secondary) arch? Or will there be two instances, one for all
> primary and one for all secondary?
>
From what I can tell, shadow Koji instances are going away entirely.
There will be one Koji system to rule them all.
>
> Will a build failure on (say) aarch64 prevent my package from
> progressing to Rawhide (x86_64), bodhi etc?
>
> -*- -*- -*-
That's what Adam Williamson seems to indicate, which I expect to be
quite problematic.
The page explicitly states it, under "Will a single arch failure affect the overall
build failure?"
3:02 p.m.
On 04/08/16 20:48, Adam Williamson wrote:
The page says that Koji will be modified to run all the per-arch
build
tasks to completion even if one fails (as opposed to how it behaves
now, cancelling all the other arch tasks as soon as any one fails), but
a failure of any of them will still constitute a failure of the overall
task.
Well that's how I read it at first as well, but if you read on it talks
about how to deal with subsequent builds seeing different libraries if
some builds had failed, which implies the task wouldn't be failed and
the builds had worked would be published.
So currently I think we can only say it's somewhat unclear what the plan
is...
Tom
--
Tom Hughes (tom(a)compton.nu)
http://compton.nu/
3:08 p.m.
On Thu, 2016-08-04 at 21:02 +0100, Tom Hughes wrote:
On 04/08/16 20:48, Adam Williamson wrote:
>
> The page says that Koji will be modified to run all the per-arch build
> tasks to completion even if one fails (as opposed to how it behaves
> now, cancelling all the other arch tasks as soon as any one fails), but
> a failure of any of them will still constitute a failure of the overall
> task.
Well that's how I read it at first as well, but if you read on it talks
about how to deal with subsequent builds seeing different libraries if
some builds had failed, which implies the task wouldn't be failed and
the builds had worked would be published.
So currently I think we can only say it's somewhat unclear what the plan
is...
It talks about that as a *justification* for not doing it:
"The issue with not failing all builds when a single arch fails is how
we deal with any builds that are dependent on that package?"
i.e. it's saying the reason they chose *not* to allow builds to succeed
with some arches failing is because of the problem of dependent
packages then being out of sync across arches.
3:16 p.m.
On Thu, Aug 4, 2016 at 4:08 PM, Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
On Thu, 2016-08-04 at 21:02 +0100, Tom Hughes wrote:
> On 04/08/16 20:48, Adam Williamson wrote:
>
> >
> > The page says that Koji will be modified to run all the per-arch build
> > tasks to completion even if one fails (as opposed to how it behaves
> > now, cancelling all the other arch tasks as soon as any one fails), but
> > a failure of any of them will still constitute a failure of the overall
> > task.
>
> Well that's how I read it at first as well, but if you read on it talks
> about how to deal with subsequent builds seeing different libraries if
> some builds had failed, which implies the task wouldn't be failed and
> the builds had worked would be published.
>
> So currently I think we can only say it's somewhat unclear what the plan
> is...
It talks about that as a *justification* for not doing it:
"The issue with not failing all builds when a single arch fails is how
we deal with any builds that are dependent on that package?"
i.e. it's saying the reason they chose *not* to allow builds to succeed
with some arches failing is because of the problem of dependent
packages then being out of sync across arches.
That's already the situation now, anyway. And we're not unique in
this. Debian does things similarly with their autobuilder/buildd
system. If anything we probably just need some way to track on a per
arch level to warn when it happens so that the right people can deal
with the situation.
--
真実はいつも一つ！/ Always, there's only one truth!
3:30 a.m.
On 08/04/2016 05:07 PM, Peter Robinson wrote:
All the details of the proposal along with FAQ have been put on a
wiki
page here[1]
so please go and read it and ask any questions that aren't answered in
the FAQ here.
For rawhide, ppc.koji is currently behind primary by two weeks (or even
more). What's the cause of these delays? Is it koji-shadow bugs or
lack of automation? I hope it's not raw builder throughput.
I know that there are koji-shadow bugs which cause build root
corruption, and the NVR sync from primary is not exact (in the sense
that packages are built against different libraries), so I hope we can
get rid of koji-shadow.
Thanks,
Florian
4:02 a.m.
On Fri, 5 Aug 2016 10:30:38 +0200
Florian Weimer <fweimer(a)redhat.com> wrote:
On 08/04/2016 05:07 PM, Peter Robinson wrote:
> All the details of the proposal along with FAQ have been put on a
> wiki page here[1]
> so please go and read it and ask any questions that aren't answered
> in the FAQ here.
For rawhide, ppc.koji is currently behind primary by two weeks (or
even more). What's the cause of these delays? Is it koji-shadow
bugs or lack of automation? I hope it's not raw builder throughput.
nothing in the infrastructure, the problem is with ruby and mariadb
build that are failing in their test-suites and we are unable to
reproduce them outside of koji. And with Flock this week there was
little to no progress on the issue. koji-shadow builds the builds in the
correct order (with same buildroots as were used in primary) it means a
missing (broken) build stops the whole process.
ruby = https://bugzilla.redhat.com/show_bug.cgi?id=1361037
mariadb is about resource exhaustion when starting threads (IIRC),
interestingly recent golang build in primary koji sees very similar
issue
I know that there are koji-shadow bugs which cause build root
corruption, and the NVR sync from primary is not exact (in the sense
koji-shadow itself shouldn't cause buildroot corruption, rather wrong
configuration change
that packages are built against different libraries), so I hope we
can get rid of koji-shadow.
and there are other reasons to get rid of koji-shadow too :-)
Dan
5:10 a.m.
> All the details of the proposal along with FAQ have been put on a
wiki
> page here[1]
> so please go and read it and ask any questions that aren't answered in
> the FAQ here.
>
> Regards,
> Peter
>
> [1] https://fedoraproject.org/wiki/Architectures/RedefiningSecondaryArchitect...
> --
What is the update for this statement:
Q: When will the new ARMv7 builders be in place?
A: Soon! The current plan is mid to late July. This proposal isn't
impacted by this as ARMv7 is already a primary architecture.
since we are past July.. is it July 2017 :)?
I am currently awaiting a new firmware to fix an issue from the
vendor. It's currently going through their QA process so it's ASAP and
I'm hoping to be dealing with it early next week.
Peter
5:22 a.m.
> We are planning to change the way Alternate Architectures (non
x86_64)
> are handled
> in terms of "primary" vs "secondary". The definition of what is
> primary or secondary
> is already handled more in terms of the build artifact outputs (images, LiveCDs,
> installers, containers etc) for i686 deliverables. As part of this redefinition
> this means that the location in "koji instances" of the rpm builds is
removed as
> a part of the definition requirement of what constitutes
> primary/secondary and the
> architectures are named "Alternate Architectures" (and Experimental
> architectures
> for the likes of MIPs/RISC-V) as opposed to primary/secondary. As a
> result of this
> change it is planned to merge the old "secondary" koji instances into a
single
> koji instance along with all the current "Primary" architectures.
>
> All the details of the proposal along with FAQ have been put on a wiki
> page here[1]
> so please go and read it and ask any questions that aren't answered in
> the FAQ here.
I do have serious concerns about the impact of this in terms of build
failures. Reading the reply to " Q: Why do I have to worry about
s390x/powerpc/aarch64 when I didn't before?", it implies there will be
no change to koji in terms of build failures: i.e. a failure on *any*
arch will cause the entire build to be failed.
There will be a slight change that a failure in an arch won't cancel
the other arches and each one will run to completion (pass/fail) but
the overall primary task will still fail.
The answer...honestly does not convince me. I think the result will
be
a combination both of an increase in failed builds and the issues
caused by them, and an increase in the number of packages which simply
disable building on an arch entirely due to a lack of will to deal with
build issues (and/or slow build time) on that arch.
The data we have for build failures across all the arches show that
not to be the case.
All the pure noarch packages, which is over 50% of the distribution,
currently can and are regularly built on ppc64/ppc64le builders now
anyway due to them being in primary koji for EPEL so for a large
percentage it's already dealing with a lot of the arches we're due to
cover anyway and there's not been a single issue reported there in the
12 months that ppc64le has been present.
And in response to the "slow built times" the builders in the non
primary koji instance are of equivalent or faster than the x86
builders. EG the ppc64 builders use to be a LOT slower than x86 back
when we had Power6 builders, but that hasn't been the case for over 3
years, and the current Power8 generation builders are actually faster
than the x86 builders. For ARMv7 (which is not part of this because
it's already in primary) will actually get faster builders on aarch64
as part of this change.
With secondary koji instances, neither of these are major issues,
and
secondary arch teams are able to work on build fixes for those arches
without the maintainers being bothered by them.
In most cases the maintainers are still bothered by failures even from
the secondary arches anyway, it will certainly be more up front to
them but the core packages that historically have issues (toolchains
etc) already have maintainers that actively test and support the non
x86 architectures anyway.
Has any consideration been given to the possibility of increasing
Koji's flexibility here, by allowing for arches to be designated as
non-fatal, so a package build failure on that arch would not cause the
task to be considered a failure?
Yes, but how would you deal with a soname bump where if it's not fatal
on an arch what happens when something is then rebuilt against one
version on one arch and a different version on a different arch. You
end up in a big mess really quickly. It ends up being a lot less work
(even in the current situation with non x86 arches) if the issue is
fixed from the outset.
Peter
5:29 a.m.
> [1]
https://fedoraproject.org/wiki/Architectures/RedefiningSecondaryArchitect...
I skimmed all this and I'm still a bit confused.
Will there be one Koji instance compiling for every (current primary +
current secondary) arch? Or will there be two instances, one for all
primary and one for all secondary?
There will be a single instance of koji for all architectures.
Will a build failure on (say) aarch64 prevent my package from
progressing to Rawhide (x86_64), bodhi etc?
Yes. a failure on one arch, just like in primary now for
x86_64/i686/armv7hl, this won't change.
On the subject of alternate architectures, I'm making available
Fedora
images available in virt-builder for aarch64, armv7l, ppc64 and
ppc64le. There is a complete set for Fedora 23, and a partial set for
Fedora 24 (booting problems on ppc64 - will be solved eventually).
I'm not sure what the question is here.
You can run these up on x86_64 hosts quite easily. For an example
of
how see:
https://rwmj.wordpress.com/2015/04/15/virt-builder-fedora-21-ppc64-and-pp...
(The virt-install method is now the recommended one. Don't run qemu
directly.)
Yes, we use virt-install for deployment of VMs already across
aarch64/ppc64/ppc64le.
On the subject of RISC-V, I'm still plugging away at this.
It's
rather slow going, but you can take a look at:
http://git.annexia.org/?p=fedora-riscv.git;a=summary
There is nothing much usable at the moment, and many stumbling blocks.
Yes, but this is an "experimental arch" and is completely out of scope
of this proposal.
2:58 p.m.
On 2016-08-05 05:29, Peter Robinson wrote:
> On the subject of RISC-V, I'm still plugging away at this.
It's
> rather slow going, but you can take a look at:
>
> http://git.annexia.org/?p=fedora-riscv.git;a=summary
>
> There is nothing much usable at the moment, and many stumbling blocks.
Yes, but this is an "experimental arch" and is completely out of scope
of this proposal.
True, but it does beg the question, how will future new arches (e.g.
riscv, or mips) be handled? Will koji-shadow still have a place in
that, and what criteria will be established for merging into primary?
--
Yaakov Selkowitz
Software Engineer - Platform Enablement Group
Red Hat, Inc.
10:02 a.m.
>> On the subject of RISC-V, I'm still plugging away at
this. It's
>> rather slow going, but you can take a look at:
>>
>> http://git.annexia.org/?p=fedora-riscv.git;a=summary
>>
>> There is nothing much usable at the moment, and many stumbling blocks.
>
>
> Yes, but this is an "experimental arch" and is completely out of scope
> of this proposal.
True, but it does beg the question, how will future new arches (e.g. riscv,
or mips) be handled? Will koji-shadow still have a place in that, and what
criteria will be established for merging into primary?
It would be a different set of requirements.
It's more similar to how architectures achieved the former "secondary"
status. These requirements basically equate to:
* availability of decent enterprise hardware that can be added to the
Fedora data centre for build systems and other requirements (rel-eng,
QA etc)
* proven commitment from a company/team/community to support the architecture
* reasonable availability of hardware. It's a lot of work for the
various Fedora teams to support any architecture, even as a secondary
architecture.
Likely a lot of other stuff that I've missed out but I think it gives
a general idea.
In the time between bring up phase and acceptance of the above
koji-shadow still remains the proper means of ensuring builds are as
close to Fedora mainline as possible.
Peter
5:37 p.m.
Peter Robinson wrote:
We are planning to change the way Alternate Architectures (non
x86_64)
are handled in terms of "primary" vs "secondary".
Let me repost here what I already posted at:
https://fedorahosted.org/fesco/ticket/1592#comment:14
There, I wrote:
| IMHO, it is entirely unacceptable to let toolchain bugs on obscure
| architectures (bugs that, in my experience, are much more frequent than
| the OP is claiming) hold our builds hostage (through the proposed "fail on
| one = fail on all" principle). It is already painful enough with ARM
| (e.g., this showstopper:
| https://bugzilla.redhat.com/show_bug.cgi?id=1342095 has been breaking
| builds of several Qt/KDE packages for months and is still not fixed – the
| only workaround that makes the affected packages build on ARM makes the
| output not Fedora-complaint (it is not allowed to require NEON)). I have
| seen even worse architecture-specific bugs and limitations (e.g. on the
| number of relocations) from targets such as ppc64 (the obscure "number of
| relocations" thing is a real ppc64 example) that this proposal would also
| make blocking for builds.
|
| IMHO, only ONE architecture (probably x86_64) should block builds. A
| failure on any other architecture (including ARM) should affect only the
| failing architecture.
Kevin Kofler
5:59 p.m.
Peter Robinson wrote:
There will be a slight change that a failure in an arch won't
cancel
the other arches and each one will run to completion (pass/fail) but
the overall primary task will still fail.
I don't see how wasting Koji resources on completing an already failed build
helps anybody.
The data we have for build failures across all the arches show that
not to be the case.
Having been plagued by obscure architecture-specific toolchain bugs several
times (see also my other mail in this thread), I don't think you are seeing
the whole picture there.
All the pure noarch packages, which is over 50% of the distribution,
Noarch packages are by their very nature unaffected by the vast majority of
architecture-specific issues. (Also because they are typically interpreted
packages where the "build" process is trivial.) They are not a
representative sample in any way.
currently can and are regularly built on ppc64/ppc64le builders now
anyway due to them being in primary koji for EPEL so for a large
percentage it's already dealing with a lot of the arches we're due to
cover anyway and there's not been a single issue reported there in the
12 months that ppc64le has been present.
Are you sure that Fedora noarch packages are actually being built on ppc64
builders? They would need a ppc64 Fedora in the buildroot for that. I
remember that back when PPC was demoted to secondary, Fedora noarch builds
stopped happening on PPC builders for the releases where PPC was no longer
primary (while still happening for the updates of the last PPC-as-primary
release), because of that buildroot thing. It was similar the other way
round when ARM was added to the primary Koji instance, only the new release
was able to use ARM builders for noarch packages.
And in response to the "slow built times" the builders in
the non
primary koji instance are of equivalent or faster than the x86
builders. EG the ppc64 builders use to be a LOT slower than x86 back
when we had Power6 builders, but that hasn't been the case for over 3
years, and the current Power8 generation builders are actually faster
than the x86 builders.
Does that also hold for build tasks that cannot be parallelized (e.g.:
custom specfile scripts, handwritten build scripts from upstream, Makefiles
that do not support %{_smp_mflags} due to race conditions, etc.)?
And is this really a strength of the new architectures? To me, it sounds
more like a sign that the x86 builders need to be renewed. Or is aarch64
really competitive at performance with x86 now?
For ARMv7 (which is not part of this because it's already in
primary) will
actually get faster builders on aarch64 as part of this change.
And making ARMv7 primary was a big mistake that ought to be corrected now
that it is clear that Fedora will never (or at least not in the foreseeable
future) run on the ARM devices 99% of ARM users out there use (smartphones),
and that the remaining ARM niche is being obsoleted by aarch64. ARMv7 should
become secondary again (with the "old" definition of "secondary", not
your
proposed one). Then you can also (instead of your proposed change) easily
put your fancy new builders on the ARM Koji instance and build both ARMv7
and aarch64 on them without bothering the x86 builds.
In most cases the maintainers are still bothered by failures even
from
the secondary arches anyway, it will certainly be more up front to
them but the core packages that historically have issues (toolchains
etc) already have maintainers that actively test and support the non
x86 architectures anyway.
Well, I care about the primary architectures mainly, and usually leave
issues on the secondary architectures to the secondary architecture
maintainer. And IMHO, that's how it should be. The people who care about
exotic niche architectures should be the ones doing the work for them.
Yes, but how would you deal with a soname bump where if it's not
fatal
on an arch what happens when something is then rebuilt against one
version on one arch and a different version on a different arch. You
end up in a big mess really quickly. It ends up being a lot less work
(even in the current situation with non x86 arches) if the issue is
fixed from the outset.
The "big mess" you describe is how Debian does things, it works well for
them. The alternative approach is how koji-shadow does things: just hold all
builds that have the failed build in the buildroot. (And the easiest way to
implement that is to just keep using koji-shadow. I don't see why we have to
shoehorn everything into the main Koji instance.)
Kevin Kofler
7:34 p.m.
On Sun, Aug 07, 2016 at 12:59:06AM +0200, Kevin Kofler wrote:
Peter Robinson wrote:
> There will be a slight change that a failure in an arch won't cancel
> the other arches and each one will run to completion (pass/fail) but
> the overall primary task will still fail.
I don't see how wasting Koji resources on completing an already failed build
helps anybody.
I think this was already mentioned in this thread, but anyway, the goal
is to be able to distinguish the case where just one architecture x fails to
build from the case where two or more architectures fail to build, and
architecture x was just the fastest.
> The data we have for build failures across all the arches show
that
> not to be the case.
Having been plagued by obscure architecture-specific toolchain bugs several
times (see also my other mail in this thread), I don't think you are seeing
the whole picture there.
Bug #1342095 is hardly earth-shattering. Simply reverting to distribution
default CFLAGS seems to work around the problem.
And please note that the plan is that by removing the need to play catch
up with koji-shadow for secondary architectures, the maintainers for those
architectures will have more time to handle actual bugs, so it's likely
that #1342095 might be handled faster in the future.
> And in response to the "slow built times" the builders
in the non
> primary koji instance are of equivalent or faster than the x86
> builders. EG the ppc64 builders use to be a LOT slower than x86 back
> when we had Power6 builders, but that hasn't been the case for over 3
> years, and the current Power8 generation builders are actually faster
> than the x86 builders.
Does that also hold for build tasks that cannot be parallelized (e.g.:
custom specfile scripts, handwritten build scripts from upstream, Makefiles
that do not support %{_smp_mflags} due to race conditions, etc.)?
Probably not. But how many packages do we have that A) are big, B) have broken
parallel build, C) are active and rebuild regularly? It'd get that A ∩ B ∩ C
isn't that big.
Zbyszek
2817
days inactive
2821
days old
22 comments
12 participants
participants (12)
-
Adam Williamson -
Dan Horák -
Florian Weimer -
Jon -
Kevin Kofler -
Neal Gompa -
Peter Robinson -
Richard W.M. Jones -
Stephen John Smoogen -
Tom Hughes -
Yaakov Selkowitz -
Zbigniew Jędrzejewski-Szmek