On 21.11.2017 11:26, Dan Horák wrote: [...]
qemu s390x emulation does not work with code compiled for z12. Would a real virtual machine be what you need? The Fedora team DOES have access to a z13. Not sure how much resources are available, but can you contact Dan Horak (on cc) if there is enough spare capacity.
Christian is right, we have a publicly accessible guest running Fedora on the Marist College z13 mainframe. It's currently used by ~5 projects (for example glibc and qemu) as their build and CI host, so adding another project depends how intensive ovirt's usage would be.
As a first step one could only build the packages needed for the KVM host. At this point in time that would be vdsm and ovirt-host, both are building rather quickly. It should be possible to ensure that only these are built on a s390 system using appropriate node filters. [...]
sending once more for the s390x list ...
On Fri, 24 Nov 2017 10:05:06 +0100 Viktor Mihajlovski mihajlov@linux.vnet.ibm.com wrote:
On 21.11.2017 11:26, Dan Horák wrote: [...]
qemu s390x emulation does not work with code compiled for z12. Would a real virtual machine be what you need? The Fedora team DOES have access to a z13. Not sure how much resources are available, but can you contact Dan Horak (on cc) if there is enough spare capacity.
Christian is right, we have a publicly accessible guest running Fedora on the Marist College z13 mainframe. It's currently used by ~5 projects (for example glibc and qemu) as their build and CI host, so adding another project depends how intensive ovirt's usage would be.
As a first step one could only build the packages needed for the KVM host. At this point in time that would be vdsm and ovirt-host, both are building rather quickly. It should be possible to ensure that only these are built on a s390 system using appropriate node filters.
I would say there are still 2 options how to build "add-on" packages that install/run on top of Fedora and it depends whether they create dependency chains or are standalone.
For standalone packages (= their Buildrequires can be resolved solely from the Fedora repos) one can use scratch builds koji and grab the results (= rpms) when the build finishes.
For packages with dependency chains is using an own "builder" machine required. The mock tool is capable of managing a repo with built rpms, so it shouldn't be difficult to achieve on our public guest.
Dan
On 24 November 2017 at 11:05, Viktor Mihajlovski mihajlov@linux.vnet.ibm.com wrote:
On 21.11.2017 11:26, Dan Horák wrote: [...]
qemu s390x emulation does not work with code compiled for z12. Would a real virtual machine be what you need? The Fedora team DOES have access to a z13. Not sure how much resources are available, but can you contact Dan Horak (on cc) if there is enough spare capacity.
Christian is right, we have a publicly accessible guest running Fedora on the Marist College z13 mainframe. It's currently used by ~5 projects (for example glibc and qemu) as their build and CI host, so adding another project depends how intensive ovirt's usage would be.
As a first step one could only build the packages needed for the KVM host. At this point in time that would be vdsm and ovirt-host, both are building rather quickly. It should be possible to ensure that only these are built on a s390 system using appropriate node filters. [...]
We can get more accurate data by looking at the ppc64c build history (We support ppc64le only for hypervisor usage, similar to what is intended for s390). Here is the history for vdsm: http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-ppc64le/buildTi... (~20 builds a day taking 1-2 minutes each) And here is the one for ovirt-host: http://jenkins.ovirt.org/job/ovirt-host_master_check-patch-el7-ppc64le/build... (only 1 build in history, taking 3-4 minutes)
Looking at what else we have building on ppc64le: http://jenkins.ovirt.org/search/?q=master_build-artifacts-el7-ppc64le I can also see ioprocess with is a vdsm dependency, and the SDK which is probably not really needed. So for ioprocess: http://jenkins.ovirt.org/job/ioprocess_master_build-artifacts-el7-ppc64le/bu... I'd say its very rarely built.
So we end up with ~20 1-2 minute builds a day (Timed but the amount of Fedora versions we want to support, but what will probably be just one), with the rest being a statistical error...
I wonder about sharing a VM with other project though. We do use mock for running the build script so the build itself should be fairly isolated, but we have some of our own wrapper scripts around mock that do things trying to keep build dependencies in the chroot cache over time. We're also incompatible with mock's new systemd-nspawn backend, so we force it to work with the older chroot-based backend. If other projects are using mock as well, I wonder if we may end up with race conditions arising from shared use of /var/lib/mock.
Bottom line - we may end up being a little noisy neighbours if we share a VM, but we can try that and see what happens, how to we move foreward with trying that?
On Fri, 24 Nov 2017 13:16:53 +0200 Barak Korren bkorren@redhat.com wrote:
On 24 November 2017 at 11:05, Viktor Mihajlovski mihajlov@linux.vnet.ibm.com wrote:
On 21.11.2017 11:26, Dan Horák wrote: [...]
qemu s390x emulation does not work with code compiled for z12. Would a real virtual machine be what you need? The Fedora team DOES have access to a z13. Not sure how much resources are available, but can you contact Dan Horak (on cc) if there is enough spare capacity.
Christian is right, we have a publicly accessible guest running Fedora on the Marist College z13 mainframe. It's currently used by ~5 projects (for example glibc and qemu) as their build and CI host, so adding another project depends how intensive ovirt's usage would be.
As a first step one could only build the packages needed for the KVM host. At this point in time that would be vdsm and ovirt-host, both are building rather quickly. It should be possible to ensure that only these are built on a s390 system using appropriate node filters. [...]
We can get more accurate data by looking at the ppc64c build history (We support ppc64le only for hypervisor usage, similar to what is intended for s390). Here is the history for vdsm: http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-ppc64le/buildTi... (~20 builds a day taking 1-2 minutes each) And here is the one for ovirt-host: http://jenkins.ovirt.org/job/ovirt-host_master_check-patch-el7-ppc64le/build... (only 1 build in history, taking 3-4 minutes)
Looking at what else we have building on ppc64le: http://jenkins.ovirt.org/search/?q=master_build-artifacts-el7-ppc64le I can also see ioprocess with is a vdsm dependency, and the SDK which is probably not really needed. So for ioprocess: http://jenkins.ovirt.org/job/ioprocess_master_build-artifacts-el7-ppc64le/bu... I'd say its very rarely built.
So we end up with ~20 1-2 minute builds a day (Timed but the amount of Fedora versions we want to support, but what will probably be just one), with the rest being a statistical error...
I wonder about sharing a VM with other project though. We do use mock for running the build script so the build itself should be fairly isolated, but we have some of our own wrapper scripts around mock that do things trying to keep build dependencies in the chroot cache over time. We're also incompatible with mock's new systemd-nspawn backend, so we force it to work with the older chroot-based backend. If other projects are using mock as well, I wonder if we may end up with race conditions arising from shared use of /var/lib/mock.
it should work fine
Bottom line - we may end up being a little noisy neighbours if we share a VM, but we can try that and see what happens, how to we move foreward with trying that?
ok, I'm pretty sure we can make it work :-) Please send me your public SSH key and preferred username, then I'll set up you an account for you and we can work on the remaining details.
Dan
On 24 November 2017 at 14:36, Dan Horák dan@danny.cz wrote:
ok, I'm pretty sure we can make it work :-) Please send me your public SSH key and preferred username, then I'll set up you an account for you and we can work on the remaining details.
I would actually like two keys to be configured, one for our staging system:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCcaW1eHhuFfPKgDcSir/D2/qZlBwMSSXUXZi4F8SOt0C6WFggRcZB6kjk73GyzkZ879Wlr97WITAPXaaEFSNnaa0TSfTpuElqhOipR/IEM9KYDDfYIIoABBebhn6kpBBQ81gd3L4+8Lv6xse6YBu4/HhfILBbUN20DVUYd9vGUc2y0c9RasJjotdg7+1iUzbT/dqPG1OX/S4M/qIF6wcygnedHt2KtPE+QosCACNdtshGHwngO9H+wXv9e/37WFwU6dRMESCxrBAM+gxO8+0nLANW28GDr5EGYNs4gy5reyTKS8qqswHqK4h5bi7Ad1BSx29DUa8wEOUO/TV2eiUsz jenkins@jenkins-staging.phx.ovirt.org
And another for the production system:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDsZ34L+B3YzL7a6zCrJB41r/IqM/s1ILXyjslApSrtquQRtUcbeoE7kS4PdyhO2U4Pu91EzYMPWc7JVnQirwKX5ksXwxZn/Y8f5KzKm5IfPRJfX6sBWS9eGRsyLj5JQjHiVYiBSsACidIr8zc3lJo/nxhp18wj5Ao4h5rhqpw/P+u53/NQ0KvRQtrBFxgWR9JM6KpcjB6rVzm1OBJQPe9aSm97NLh3ijXxYNrbIpXt/YoyByP36QVlcM+L9idFAWY2TkCX5mWclCJJeCint9+SxD0gRW3/tgNWwxx7nkFDGdl/WKhgT0JjmCVFqSG/cGNYYMX+A25zKqqD1SqPNFhx jenkins@jenkins.phx.ovirt.org
The user names for the systems are 'jenkins-staging' and 'jenkins' respectively, though it'll also be ok to use a different user name and the same user for both. Both those users need to belong to the 'mock' group.
I have some places in the code where it just assumes it's got password-less sudo configured and tries to install some needed dependencies on its own. Those dependencies are most probably not needed for the builds we're discussing here (They include libvirt and docker that are used by build processes that escape mock to generate container and VM images...) , but I'll need to add some checking in my code and skip those settings if sudo permissions are not available. This is why I'd rather connect the staging system first.
On 24 November 2017 at 14:36, Dan Horák dan@danny.cz wrote:
On Fri, 24 Nov 2017 13:16:53 +0200 Barak Korren bkorren@redhat.com wrote:
On 24 November 2017 at 11:05, Viktor Mihajlovski mihajlov@linux.vnet.ibm.com wrote:
On 21.11.2017 11:26, Dan Horák wrote: [...]
qemu s390x emulation does not work with code compiled for z12. Would a real virtual machine be what you need? The Fedora team DOES have access to a z13. Not sure how much resources are available, but can you contact Dan Horak (on cc) if there is enough spare capacity.
Christian is right, we have a publicly accessible guest running Fedora on the Marist College z13 mainframe. It's currently used by ~5 projects (for example glibc and qemu) as their build and CI host, so adding another project depends how intensive ovirt's usage would be.
As a first step one could only build the packages needed for the KVM host. At this point in time that would be vdsm and ovirt-host, both are building rather quickly. It should be possible to ensure that only these are built on a s390 system using appropriate node filters. [...]
We can get more accurate data by looking at the ppc64c build history (We support ppc64le only for hypervisor usage, similar to what is intended for s390). Here is the history for vdsm: http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-ppc64le/buildTi... (~20 builds a day taking 1-2 minutes each) And here is the one for ovirt-host: http://jenkins.ovirt.org/job/ovirt-host_master_check-patch-el7-ppc64le/build... (only 1 build in history, taking 3-4 minutes)
Looking at what else we have building on ppc64le: http://jenkins.ovirt.org/search/?q=master_build-artifacts-el7-ppc64le I can also see ioprocess with is a vdsm dependency, and the SDK which is probably not really needed. So for ioprocess: http://jenkins.ovirt.org/job/ioprocess_master_build-artifacts-el7-ppc64le/bu... I'd say its very rarely built.
So we end up with ~20 1-2 minute builds a day (Timed but the amount of Fedora versions we want to support, but what will probably be just one), with the rest being a statistical error...
I wonder about sharing a VM with other project though. We do use mock for running the build script so the build itself should be fairly isolated, but we have some of our own wrapper scripts around mock that do things trying to keep build dependencies in the chroot cache over time. We're also incompatible with mock's new systemd-nspawn backend, so we force it to work with the older chroot-based backend. If other projects are using mock as well, I wonder if we may end up with race conditions arising from shared use of /var/lib/mock.
it should work fine
Bottom line - we may end up being a little noisy neighbours if we share a VM, but we can try that and see what happens, how to we move foreward with trying that?
ok, I'm pretty sure we can make it work :-) Please send me your public SSH key and preferred username, then I'll set up you an account for you and we can work on the remaining details.
An update for everyone woh may have been watching this thread - we made it work.
With Dan's kind help we've attached an s390x VM to oVirt's CI infrastructure. I've then gone ahead and made some code changes to make our CI code play nice on it (So far we just assumed we own the execution slaves and can do what we want on them...). Following that I've gone ahead and added the basic configuration needed to make the oVirt CI system support s390x jobs.
For now we only support using Fedora 26 on s390x. Please let me know if other distributions are desired.
The code changes I've made had already been tested and are now pending code review: https://gerrit.ovirt.org/c/85219 https://gerrit.ovirt.org/c/85221
Once those patches are merged it will become possible to add s390x jobs to any oVirt project by adding '390x' to the list of architectures targeted by the project in the JJB YAML, as well as setting the 'node_filter' to be 's390x' for that architecture.
On 10.12.2017 10:10, Barak Korren wrote: [...]
An update for everyone woh may have been watching this thread - we made it work.
What a nice surprise after returning from a few days off. Big thanks to both of you, Barak and Dan.
With Dan's kind help we've attached an s390x VM to oVirt's CI infrastructure. I've then gone ahead and made some code changes to make our CI code play nice on it (So far we just assumed we own the execution slaves and can do what we want on them...). Following that I've gone ahead and added the basic configuration needed to make the oVirt CI system support s390x jobs.
For now we only support using Fedora 26 on s390x. Please let me know if other distributions are desired.
Lately, I've been using Fedora 27 for the s390x porting, mostly because the newer package levels of the key virtualization components. If possible, Fedora 27 would be nice.
The code changes I've made had already been tested and are now pending code review: https://gerrit.ovirt.org/c/85219 https://gerrit.ovirt.org/c/85221
Once those patches are merged it will become possible to add s390x jobs to any oVirt project by adding '390x' to the list of architectures targeted by the project in the JJB YAML, as well as setting the 'node_filter' to be 's390x' for that architecture.
Looking forward to see this merged. Once the s390x RPMs show up in the repository, I will try to do a clean setup of a s390x cluster.
On Thu, 14 Dec 2017 11:24:27 +0100 Viktor Mihajlovski mihajlov@linux.vnet.ibm.com wrote:
On 10.12.2017 10:10, Barak Korren wrote: [...]
An update for everyone woh may have been watching this thread - we made it work.
What a nice surprise after returning from a few days off. Big thanks to both of you, Barak and Dan.
With Dan's kind help we've attached an s390x VM to oVirt's CI infrastructure. I've then gone ahead and made some code changes to make our CI code play nice on it (So far we just assumed we own the execution slaves and can do what we want on them...). Following that I've gone ahead and added the basic configuration needed to make the oVirt CI system support s390x jobs.
For now we only support using Fedora 26 on s390x. Please let me know if other distributions are desired.
Lately, I've been using Fedora 27 for the s390x porting, mostly because the newer package levels of the key virtualization components. If possible, Fedora 27 would be nice.
The Marist College agrees to provide us more guests, so we shouldn't see capacity issues in the near (and mid) future. F-26 is good enough for your F-27 builds, because you use mock, right?
The code changes I've made had already been tested and are now pending code review: https://gerrit.ovirt.org/c/85219 https://gerrit.ovirt.org/c/85221
Once those patches are merged it will become possible to add s390x jobs to any oVirt project by adding '390x' to the list of architectures targeted by the project in the JJB YAML, as well as setting the 'node_filter' to be 's390x' for that architecture.
Looking forward to see this merged. Once the s390x RPMs show up in the repository, I will try to do a clean setup of a s390x cluster.
Dan
On 14.12.2017 11:34, Dan Horák wrote:
On Thu, 14 Dec 2017 11:24:27 +0100
[...]
Lately, I've been using Fedora 27 for the s390x porting, mostly because the newer package levels of the key virtualization components. If possible, Fedora 27 would be nice.
The Marist College agrees to provide us more guests, so we shouldn't see capacity issues in the near (and mid) future. F-26 is good enough for your F-27 builds, because you use mock, right?
Right, a mock config for f27-s390x should do trick even on f26 as a build host. [...]
On 14 December 2017 at 12:34, Dan Horák dan@danny.cz wrote:
On Thu, 14 Dec 2017 11:24:27 +0100 Viktor Mihajlovski mihajlov@linux.vnet.ibm.com wrote:
On 10.12.2017 10:10, Barak Korren wrote: [...]
An update for everyone woh may have been watching this thread - we made it work.
What a nice surprise after returning from a few days off. Big thanks to both of you, Barak and Dan.
With Dan's kind help we've attached an s390x VM to oVirt's CI infrastructure. I've then gone ahead and made some code changes to make our CI code play nice on it (So far we just assumed we own the execution slaves and can do what we want on them...). Following that I've gone ahead and added the basic configuration needed to make the oVirt CI system support s390x jobs.
For now we only support using Fedora 26 on s390x. Please let me know if other distributions are desired.
Lately, I've been using Fedora 27 for the s390x porting, mostly because the newer package levels of the key virtualization components. If possible, Fedora 27 would be nice.
The Marist College agrees to provide us more guests, so we shouldn't see capacity issues in the near (and mid) future. F-26 is good enough for your F-27 builds, because you use mock, right?
Yeah, when I said I only support fc26, it was because I didn`t bother enabling the mock configuration for fc27 on s390x on our system.
I'll try to find some time to do that later today.
-
Barak Korren -
Dan Horák -
Viktor Mihajlovski