Upstream tip wanted: CI service for Big Endian acrhes

[Miro Hrončok]

Recently I've reported some Big Endian related test failures to an 
upstream project [0].

I was asked by an upstream project maintainer, whether I know some free 
Continuous Integration services where they can easily run their 
testsuite on Big Endian.

Any tips?

* Upstream uses Travis CI to test on x86_64 Linux (Ubuntu)
  * Upstream uses AppVeyor to test on Microsoft Windows
  * It's a pure Python project, noarch, but some changes need to be done 
when loading/saving binary data (LE) with NumPy on BE system.

What I've considered:

* COPR (but there is no big endian arch)
  * (Ab)using Koji (I guess that would be considered a bad practice?)
  * using QUEMU on Travis CI [1]

Any better tips? Thanks

[0] https://github.com/mikedh/trimesh/issues/249
[1] 
https://developer.ibm.com/linuxonpower/2017/07/28/travis-multi-architecture-...

[Miroslav Suchý]

Dne 12. 11. 18 v 13:32 Miro Hrončok napsal(a):
> * COPR (but there is no big endian arch)
>  * (Ab)using Koji (I guess that would be considered a bad practice?)
>  * using QUEMU on Travis CI [1]

Use Mock builds (on Travis?) with --forcearch. See
https://github.com/rpm-software-management/mock/wiki/Release-Notes-1.4.11

Miroslav

[Jerry James]

On Mon, Nov 12, 2018 at 8:19 AM Miroslav Suchý  wrote:
> Use Mock builds (on Travis?) with --forcearch. See
> https://github.com/rpm-software-management/mock/wiki/Release-Notes-1.4.11

If I try to use mock with --forcearch s390x on my x86_64 box, and the
buildroot pulls in gdk-pixbuf2, then installation of the packages
hangs.  I don't know where the problem lies, but there's a bug
somewhere in there.  See
https://lists.fedoraproject.org/archives/list/[e-mail]/...

[Dan Horák]

On Mon, 12 Nov 2018 13:32:57 +0100
Miro Hrončok  wrote:
> Recently I've reported some Big Endian related test failures to an 
> upstream project [0].
>
> I was asked by an upstream project maintainer, whether I know some
> free Continuous Integration services where they can easily run their 
> testsuite on Big Endian.
>
> Any tips?
> Upstream uses Travis CI to test on x86_64 Linux (Ubuntu)
> Upstream uses AppVeyor to test on Microsoft Windows
> It's a pure Python project, noarch, but some changes need to be
>
> done when loading/saving binary data (LE) with NumPy on BE system.
>
> What I've considered:
> COPR (but there is no big endian arch)
> (Ab)using Koji (I guess that would be considered a bad practice?)
> using QUEMU on Travis CI [1]

I have seen some other projects using user-space QEMU in Travis for
checking eg. armv7 builds, but don't ask me about the names :-)

One problem is that these days it is mainly s390x being an actively
developed big endian architecture in Linux as Power on Linux is focusing
on ppc64le. But there is the OpenMainframe project [1] backed by Linux
Foundation, so in the long term we might find help there.

Dan

[1] https://www.openmainframeproject.org/

[Josh Stone]

On 11/12/18 6:57 AM, Dan Horák wrote:
> On Mon, 12 Nov 2018 13:32:57 +0100
> Miro Hrončok  wrote:
> > Recently I've reported some Big Endian related test failures to an 
> > upstream project [0].
> >
> > I was asked by an upstream project maintainer, whether I know some
> > free Continuous Integration services where they can easily run their 
> > testsuite on Big Endian.
> >
> > Any tips?
> > Upstream uses Travis CI to test on x86_64 Linux (Ubuntu)
> > Upstream uses AppVeyor to test on Microsoft Windows
> > It's a pure Python project, noarch, but some changes need to be
> >
> > done when loading/saving binary data (LE) with NumPy on BE system.
> >
> > What I've considered:
> > COPR (but there is no big endian arch)
> > (Ab)using Koji (I guess that would be considered a bad practice?)
> > using QUEMU on Travis CI [1]
>
> I have seen some other projects using user-space QEMU in Travis for
> checking eg. armv7 builds, but don't ask me about the names :-)

Rust does this.  The actual builds are still cross-compiled, but QEMU is
used to execute ARM tests.
> One problem is that these days it is mainly s390x being an actively
> developed big endian architecture in Linux as Power on Linux is focusing
> on ppc64le. But there is the OpenMainframe project [1] backed by Linux
> Foundation, so in the long term we might find help there.
> Dan
>
> [1] https://www.openmainframeproject.org/
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Jun Aruga]

I just created the topic on Travis community page.
https://travis-ci.community/t/multiarch-testing-tips/862

Jun

[Jun Aruga]

For this old topic running CI services for Big endian arches, I just
share the current tips.

Until recently Travis CI with the native s390x was a candidate. Though
it's not really free for every open source project,  if your project
can pay the fee, it might be a candidate.
https://blog.travis-ci.com/2019-11-12-multi-cpu-architecture-ibm-power-ibm-z

Cirrus CI persistent workers feature to enable the CI on your own self
servers as a CI running host.
The upstream Ruby project is considering that they will try this way.
https://github.com/cirruslabs/cirrus-ci-docs/issues/263#issuecomment-7469008...
https://medium.com/cirruslabs/announcing-public-beta-of-cirrus-ci-persistent...
https://cirrus-ci.org/guide/persistent-workers/

If your project can pay the server fee, you can try s390x VM from a
free trial. Then you might be run the VM with above Cirrus CI
persistent workers/
https://developer.ibm.com/components/ibm-linuxone/gettingstarted/

On Tue, Nov 13, 2018 at 2:46 PM Jun Aruga  wrote:
> I just created the topic on Travis community page.
> https://travis-ci.community/t/multiarch-testing-tips/862
>
> Jun

[Randy Barlow]

On Mon, 2018-11-12 at 13:32 +0100, Miro Hrončok wrote:
> Recently I've reported some Big Endian related test failures to an 
> upstream project [0].
>
> I was asked by an upstream project maintainer, whether I know some
> free 
> Continuous Integration services where they can easily run their 
> testsuite on Big Endian.
>
> Any tips?
> Upstream uses Travis CI to test on x86_64 Linux (Ubuntu)
> Upstream uses AppVeyor to test on Microsoft Windows
> It's a pure Python project, noarch, but some changes need to be
>
> done 
> when loading/saving binary data (LE) with NumPy on BE system.

Hi Miro!

I asked Brian Stinson about this and he said that there are PPC big-
endian VMs available in CentOS CI. I've been using CentOS CI for a
while with Bodhi and I'm happy with it.

[Dan Horák]

On Mon, 12 Nov 2018 11:02:45 -0500
Randy Barlow  wrote:
> On Mon, 2018-11-12 at 13:32 +0100, Miro Hrončok wrote:
> > Recently I've reported some Big Endian related test failures to an 
> > upstream project [0].
> >
> > I was asked by an upstream project maintainer, whether I know some
> > free 
> > Continuous Integration services where they can easily run their 
> > testsuite on Big Endian.
> >
> > Any tips?
> > Upstream uses Travis CI to test on x86_64 Linux (Ubuntu)
> > Upstream uses AppVeyor to test on Microsoft Windows
> > It's a pure Python project, noarch, but some changes need to be
> >
> > done 
> > when loading/saving binary data (LE) with NumPy on BE system.
>
> Hi Miro!
>
> I asked Brian Stinson about this and he said that there are PPC big-
> endian VMs available in CentOS CI. I've been using CentOS CI for a
> while with Bodhi and I'm happy with it.

don't forget there is no Fedora for big endian PPC any more, last was
F-28

Dan

[Brian Stinson]

On Nov 12 13:32, Miro Hrončok wrote:
> Recently I've reported some Big Endian related test failures to an upstream
> project [0].
>
> I was asked by an upstream project maintainer, whether I know some free
> Continuous Integration services where they can easily run their testsuite on
> Big Endian.
>
> Any tips?
> Upstream uses Travis CI to test on x86_64 Linux (Ubuntu)
> Upstream uses AppVeyor to test on Microsoft Windows
> It's a pure Python project, noarch, but some changes need to be done when
>
> loading/saving binary data (LE) with NumPy on BE system.
>
> What I've considered:
> COPR (but there is no big endian arch)
> (Ab)using Koji (I guess that would be considered a bad practice?)
> using QUEMU on Travis CI [1]
>
> Any better tips? Thanks
>
> [0] https://github.com/mikedh/trimesh/issues/249
> [1] https://developer.ibm.com/linuxonpower/2017/07/28/travis-multi-architecture-...
>
> -- 
> Miro Hrončok
> --
> Phone: [phone]
> IRC: mhroncok
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

This may not be helpful if you need to target S390x specifically, but we
have PPC Big-endian CentOS VMs for projects in CentOS CI:

https://wiki.centos.org/QaWiki/CI/Multiarch

I'm happy to chat with you and your upstream about providing some
resources here.

--
Brian Stinson
CentOS CI Infrastructure Team

[Jun Aruga]

Travis Ci (or any CI) + multiarch (QEMU) project + Big-endian arch
enables testing on big-endian arch.
As far as I know, this is the easiest way to test big-endian on the CI.

```
$ docker run --rm --privileged multiarch/qemu-user-static:register --reset

$ docker run --rm -t multiarch/fedora:25-ppc64 uname -a
Linux 870bdcfe1bff 4.16.15-300.fc28.x86_64 #1 SMP Tue Jun 12 00:42:35
UTC 2018 ppc64 ppc64 ppc64 GNU/Linux

$ docker run --rm -t multiarch/debian-debootstrap:s390x-jessie uname -a
Linux e774bfd6a08e 4.13.0-1008-gcp #11-Ubuntu SMP Thu Jan 25 11:08:44
UTC 2018 s390x GNU/Linux
```

See also my experiment for the topic:
https://github.com/junaruga/ci-multi-arch-test
https://github.com/junaruga/ci-multi-arch-test/blob/master/.travis.yml

Maybe below mutlarch container images for Fedora and CentOS could be
maintained with better way and newer version.
multiarch Fedora: https://hub.docker.com/r/multiarch/fedora/tags/
multiarch CentOS: https://hub.docker.com/r/multiarch/centos/tags/

Jun

On Mon, Nov 12, 2018 at 5:09 PM, Brian Stinson  wrote:
> On Nov 12 13:32, Miro Hrončok wrote:
> > Recently I've reported some Big Endian related test failures to an upstream
> > project [0].
> >
> > I was asked by an upstream project maintainer, whether I know some free
> > Continuous Integration services where they can easily run their testsuite on
> > Big Endian.
> >
> > Any tips?
> > Upstream uses Travis CI to test on x86_64 Linux (Ubuntu)
> > Upstream uses AppVeyor to test on Microsoft Windows
> > It's a pure Python project, noarch, but some changes need to be done when
> >
> > loading/saving binary data (LE) with NumPy on BE system.
> >
> > What I've considered:
> > COPR (but there is no big endian arch)
> > (Ab)using Koji (I guess that would be considered a bad practice?)
> > using QUEMU on Travis CI [1]
> >
> > Any better tips? Thanks
> >
> > [0] https://github.com/mikedh/trimesh/issues/249
> > [1] https://developer.ibm.com/linuxonpower/2017/07/28/travis-multi-architecture-...
> >
> > --
> > Miro Hrončok
> > --
> > Phone: [phone]
> > IRC: mhroncok
> > _______________________________________________
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
>
> This may not be helpful if you need to target S390x specifically, but we
> have PPC Big-endian CentOS VMs for projects in CentOS CI:
>
> https://wiki.centos.org/QaWiki/CI/Multiarch
>
> I'm happy to chat with you and your upstream about providing some
> resources here.
>
> --
> Brian Stinson
> CentOS CI Infrastructure Team
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]