Hi,
I am having troubles rebuilding Fedora packages that should use a modified ExclusiveArch set via an updated macro that's redefined by a package/build in my copr [1].
The reason for this exercise is that regular qt5-qtwebengine package can't be built on ppc64le, because the qtwebengine projects contains chromium under the covers and Google will unlikely ever merge the ppc64le support into chromium, although the port exists and is maintained out of the tree. To be able to provide packages that require qtwebengine I am building an updated qt5-qtwebengine in my copr [1] and would like to be able rebuild the other packages without further modifications, just modifying the qt5_qtwebengine_arches macro and rebuilding directly from dist-git branch.
The problem is that any(?) build in copr is first recreating the srpm on a system that doesn't include the modified qt5 (qt5-srpm-macros) package with the updated qt5_qtwebengine_arches macro and the build is "skipped" (seems I have deleted the "skipped" builds, but can be easily reproduced).
Is there a way to tell COPR to rebuild the srpm on the ppc64le builder with my copr enabled (which is ppc64le-only)? Not only it fails when building from dist-git, but even when a srpm built on host with the macro updated is submitted.
Is there a way to "force" a rebuild even the copr backend thinks there is no valid arch?
Shall I extend the copr to all arches so the modified macro exists on eg. x86_64? Is there another option?
[1] https://copr.fedorainfracloud.org/coprs/sharkcz/talos/
Thanks,
Dan
On 30. 10. 23 12:12, Dan Horák wrote:
Hi,
I am having troubles rebuilding Fedora packages that should use a modified ExclusiveArch set via an updated macro that's redefined by a package/build in my copr [1].
... The problem is that any(?) build in copr is first recreating the srpm on a system that doesn't include the modified qt5 (qt5-srpm-macros) package with the updated qt5_qtwebengine_arches macro and the build is "skipped" (seems I have deleted the "skipped" builds, but can be easily reproduced).
That is because the SRPMs are always rebuilt on the builder which sues Fedora 38. Even when building from uploaded SRPMs.
Is there a way to tell COPR to rebuild the srpm on the ppc64le builder with my copr enabled (which is ppc64le-only)? Not only it fails when building from dist-git, but even when a srpm built on host with the macro updated is submitted.
I don't think that's possible. See also https://github.com/fedora-copr/copr/issues/2900#issuecomment-1710260138 and further.
Is there a way to "force" a rebuild even the copr backend thinks there is no valid arch?
Shall I extend the copr to all arches so the modified macro exists on eg. x86_64? Is there another option?
I am afraid this won't help.
On pondělí 30. října 2023 18:33:27 CET Miro Hrončok wrote:
On 30. 10. 23 12:12, Dan Horák wrote:
Hi,
I am having troubles rebuilding Fedora packages that should use a modified ExclusiveArch set via an updated macro that's redefined by a package/build in my copr [1].
... The problem is that any(?) build in copr is first recreating the srpm on a system that doesn't include the modified qt5 (qt5-srpm-macros) package with the updated qt5_qtwebengine_arches macro and the build is "skipped" (seems I have deleted the "skipped" builds, but can be easily reproduced).
That is because the SRPMs are always rebuilt on the builder which sues Fedora 38. Even when building from uploaded SRPMs.
Is there a way to tell COPR to rebuild the srpm on the ppc64le builder with my copr enabled (which is ppc64le-only)? Not only it fails when building from dist-git, but even when a srpm built on host with the macro updated is submitted.
I don't think that's possible. See also https://github.com/fedora-copr/copr/issues/2900#issuecomment-1710260138 and further.
Is there a way to "force" a rebuild even the copr backend thinks there is no valid arch?
Shall I extend the copr to all arches so the modified macro exists on eg. x86_64? Is there another option?
I am afraid this won't help.
I agree with the points Miro made; he knows this problem best. Koji is in a comfortable situation here because each buildSRPMFromTask is done to build for a single distribution version (Fedora N). It's easy to decide which chroots to skip "in advance". In Copr, we do a similar thing, but the SRPM build serves as the base for builds in multiple distribution versions and even families (EPEL, Fedora, ...).
The design of the Exclu*Arch macros doesn't seem to account for this. These macros are distro-specific and baked into the shipped packages, potentially having different values in every distro version. It would be beneficial if we had access to these macros in the repository metadata. This way, we could read them and account for their cross-distro differences without the need to install every single distro tree we potentially want to build against, or skip builds as needed. Ideally, having all these macros baked into some kind of registry (Git?) would make them easy to download. Should we consider experimenting with a solution like this?
Another approach is to take a step back and initiate the build in every chroot, which means allocating builder resources for all the chroots. Then, we can decide "later" whether to skip the build or not. However, this approach can be very uneconomical, especially for s390x chroots this might cause slowdown because of the limited builder resources.
Pavel
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________ buildsys mailing list -- buildsys@lists.fedoraproject.org To unsubscribe send an email to buildsys-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/buildsys@lists.fedoraproject.o... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Tue, 31 Oct 2023 08:58:52 +0100 Pavel Raiskup praiskup@redhat.com wrote:
On pondělí 30. října 2023 18:33:27 CET Miro Hrončok wrote:
On 30. 10. 23 12:12, Dan Horák wrote:
Hi,
I am having troubles rebuilding Fedora packages that should use a modified ExclusiveArch set via an updated macro that's redefined by a package/build in my copr [1].
... The problem is that any(?) build in copr is first recreating the srpm on a system that doesn't include the modified qt5 (qt5-srpm-macros) package with the updated qt5_qtwebengine_arches macro and the build is "skipped" (seems I have deleted the "skipped" builds, but can be easily reproduced).
That is because the SRPMs are always rebuilt on the builder which sues Fedora 38. Even when building from uploaded SRPMs.
Is there a way to tell COPR to rebuild the srpm on the ppc64le builder with my copr enabled (which is ppc64le-only)? Not only it fails when building from dist-git, but even when a srpm built on host with the macro updated is submitted.
I don't think that's possible. See also https://github.com/fedora-copr/copr/issues/2900#issuecomment-1710260138 and further.
Is there a way to "force" a rebuild even the copr backend thinks there is no valid arch?
Shall I extend the copr to all arches so the modified macro exists on eg. x86_64? Is there another option?
I am afraid this won't help.
I agree with the points Miro made; he knows this problem best. Koji is in a comfortable situation here because each buildSRPMFromTask is done to build for a single distribution version (Fedora N). It's easy to decide which chroots to skip "in advance". In Copr, we do a similar thing, but the SRPM build serves as the base for builds in multiple distribution versions and even families (EPEL, Fedora, ...).
thanks for replies, it makes sense
The design of the Exclu*Arch macros doesn't seem to account for this. These macros are distro-specific and baked into the shipped packages, potentially having different values in every distro version. It would be beneficial if we had access to these macros in the repository metadata. This way, we could read them and account for their cross-distro differences without the need to install every single distro tree we potentially want to build against, or skip builds as needed. Ideally, having all these macros baked into some kind of registry (Git?) would make them easy to download. Should we consider experimenting with a solution like this?
this sounds pretty complex, I was more thinking about being able to specify which of my chroots to use to (re)build the srpm for my build. Would this be possible to implement?
Another approach is to take a step back and initiate the build in every chroot, which means allocating builder resources for all the chroots. Then, we can decide "later" whether to skip the build or not. However, this approach can be very uneconomical, especially for s390x chroots this might cause slowdown because of the limited builder resources.
What I am trying to avoid is to have to fork every package, modify its spec, push and rebuild. It works, but doesn't scale.
Dan
On úterý 31. října 2023 10:08:35 CET Dan Horák wrote:
On Tue, 31 Oct 2023 08:58:52 +0100 Pavel Raiskup praiskup@redhat.com wrote:
On pondělí 30. října 2023 18:33:27 CET Miro Hrončok wrote:
On 30. 10. 23 12:12, Dan Horák wrote:
Hi,
I am having troubles rebuilding Fedora packages that should use a modified ExclusiveArch set via an updated macro that's redefined by a package/build in my copr [1].
... The problem is that any(?) build in copr is first recreating the srpm on a system that doesn't include the modified qt5 (qt5-srpm-macros) package with the updated qt5_qtwebengine_arches macro and the build is "skipped" (seems I have deleted the "skipped" builds, but can be easily reproduced).
That is because the SRPMs are always rebuilt on the builder which sues Fedora 38. Even when building from uploaded SRPMs.
Is there a way to tell COPR to rebuild the srpm on the ppc64le builder with my copr enabled (which is ppc64le-only)? Not only it fails when building from dist-git, but even when a srpm built on host with the macro updated is submitted.
I don't think that's possible. See also https://github.com/fedora-copr/copr/issues/2900#issuecomment-1710260138 and further.
Is there a way to "force" a rebuild even the copr backend thinks there is no valid arch?
Shall I extend the copr to all arches so the modified macro exists on eg. x86_64? Is there another option?
I am afraid this won't help.
I agree with the points Miro made; he knows this problem best. Koji is in a comfortable situation here because each buildSRPMFromTask is done to build for a single distribution version (Fedora N). It's easy to decide which chroots to skip "in advance". In Copr, we do a similar thing, but the SRPM build serves as the base for builds in multiple distribution versions and even families (EPEL, Fedora, ...).
thanks for replies, it makes sense
The design of the Exclu*Arch macros doesn't seem to account for this. These macros are distro-specific and baked into the shipped packages, potentially having different values in every distro version. It would be beneficial if we had access to these macros in the repository metadata. This way, we could read them and account for their cross-distro differences without the need to install every single distro tree we potentially want to build against, or skip builds as needed. Ideally, having all these macros baked into some kind of registry (Git?) would make them easy to download. Should we consider experimenting with a solution like this?
this sounds pretty complex, I was more thinking about being able to specify which of my chroots to use to (re)build the srpm for my build. Would this be possible to implement?
Yes, but not easily. We don't use Mock environment right now for this task (parsing specfile Exclu*Arch values). We do it on Host right now.
This isn't solution for everyone (we basically declare that we have to split one build into multiple builds, when building for several incompatible chroots), but I agree it it might be helpful. I'd say that patches are welcome.
Another approach is to take a step back and initiate the build in every chroot, which means allocating builder resources for all the chroots. Then, we can decide "later" whether to skip the build or not. However, this approach can be very uneconomical, especially for s390x chroots this might cause slowdown because of the limited builder resources.
What I am trying to avoid is to have to fork every package, modify its spec, push and rebuild. It works, but doesn't scale.
Indeed, this is painful. Not really convenient work-around is to use the custom method, do:
copr-distgit-client clone "$package" --dist-git fedora cd "$package" || exit 1 .. patch-out or adjust all the Exclu*Arch statements .. copr-distgit-client sources # download sources copr-distgit-client srpm --outputdir . bsdtar xf /tmp/${package}*.src.rpm -C "$COPR_RESULTDIR"
Pavel
Dan
buildsys@lists.fedoraproject.org
-
Dan Horák -
Miro Hrončok -
Pavel Raiskup