ngompa opened a new pull-request against the project: `pungi-fedora` that you are following: `` Switch Fedora Cloud and Container builds to use kiwibuild ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
ngompa commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` I don't know if I'm doing this right, I would certainly appreciate some real reviews here...
CC: @adamwill @davdunc @dcavalca ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
adamwill commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` This will need at least https://pagure.io/pungi/pull-request/1735 , I believe, to work. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
adamwill commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` We also worked out that the `factory-parameters` dicts do nothing for kiwi builds. how those work is:
* pungi writes them into a config file it passes to koji - e.g. in [today's compose](https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20240306.n...), pungi ran `koji --profile=compose_koji image-build --config=/mnt/koji/compose/rawhide/Fedora-Rawhide-20240306.n.0/work/image-build/Container/docker_Fedora-Container-Base_aarch64-ppc64le-s390x-x86_64.cfg --wait`. That file can be seen [here](https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20240306.n...), and you can see it has a `[factory-parameters]` section * The Koji code that handles the `image-build` subcommand [reads that config file and passes the factory-parameters values into the options for the task](https://pagure.io/koji/blob/7d126abe91e91eed75ad3e6e995e4b0e3eccf01c/f/cli/k...) as a list called `task_options.factory_parameter` * That code then calls `_build_image_oz()` with those task options, which creates a `buildImageOz` instance with the `factory_parameter` opt list in its `opts` * `buildImageOz` ([here](https://pagure.io/koji/blob/7d126abe91e91eed75ad3e6e995e4b0e3eccf01c/f/kojih...)) passes the opts through to `make_task`, which...does a lot of stuff! It all gets complicated here and I am not gonna trace it all the way out, but it's clear that we wind up with a `BaseImageTask` ([here](https://pagure.io/koji/blob/7d126abe91e91eed75ad3e6e995e4b0e3eccf01c/f/build...)), which has a `_mergeFactoryParams()` method that actually reads in the factory parameters. It's called by `_buildBase()` and `_do_target_image()` methods, which pass them on to ImageFactory methods (`imgfac.BuildDispatcher.builder_for_base_image()` or `imgfac.BuildDispatcher.builder_for_target_image()`)
Point being, all of this is completely oz/ImageFactory-specific code paths.
Looking into how this works on the Kiwi codepath, well, this `factory-parameters` stuff just is not handled at any level on that codepath. The Pungi `kiwibuild` phase definition does not do anything with it. The Koji kiwi plugin's CLI command parser does not read config files or do anything at all with the word 'factory' in it. The Kiwi image build tasks similarlly don't.
But also, uh...it means this PR is actually wrong in another way. We can't have the variant lists be lists of dicts with `kiwibuild` and optional `factory-parameters` keys like they are in this PR, that is not what the `kiwibuild` phase expects. It's one level too much nesting. If you look at [the config @lsedlar used for the test compose](https://pagure.io/pungi/pull-request/1720#comment-199218), it doesn't look like that. The variant lists need to just be lists of image dicts. So it should be something like:
kiwibuild = { '^Cloud$': [ { 'kiwi_profile': 'Cloud-Base-Generic', 'arches': ['aarch64', 'ppc64le', 'x86_64'], 'repos': ['Everything'], 'subvariant': 'Cloud_Base', 'failable': ['ppc64le'], }, { 'kiwi_profile': 'Cloud-Base-AmazonEC2', 'arches': ['aarch64', 'x86_64'], 'repos': ['Everything'], 'subvariant': 'Cloud_Base', 'failable': ['*'], },
rather than the way it is now. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
adamwill commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` This also needs https://pagure.io/pungi/pull-request/1736 (thanks @jcline ). ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
ngompa commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` All the requested changes have been made. Corresponding changes have been made to fedora-kiwi-descriptions too. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
adamwill commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` So, what's the plan here? Should we ask @lsedlar to do a release and build of Pungi, then merge this for Rawhide and see what happens? ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
kevin commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` I think getting rawhide to work would be best and much easier than trying to do a complete test compose... but if folks really want to do that, we will need to create a fedora-test-compose.conf pungi config for it...
``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
ngompa commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` Let's just start with Rawhide. Once it's good, I'll branch kiwi-descriptions for f40 and make the corresponding PR. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
kevin commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` seeing some validation errors...
``` Failed validation in kiwibuild.^Cloud$.0: 'description_scm' is a required property Failed validation in kiwibuild.^Cloud$.0: 'description_path' is a required property Failed validation in kiwibuild.^Cloud$.1: 'description_scm' is a required property Failed validation in kiwibuild.^Cloud$.1: 'description_path' is a required property Failed validation in kiwibuild.^Cloud$.2: 'description_scm' is a required property Failed validation in kiwibuild.^Cloud$.2: 'description_path' is a required property Failed validation in kiwibuild.^Cloud$.3: 'description_scm' is a required property Failed validation in kiwibuild.^Cloud$.3: 'description_path' is a required property Failed validation in kiwibuild.^Cloud$.4: 'description_scm' is a required property Failed validation in kiwibuild.^Cloud$.4: 'description_path' is a required property Failed validation in kiwibuild.^Cloud$.5: 'description_scm' is a required property Failed validation in kiwibuild.^Cloud$.5: 'description_path' is a required property Failed validation in kiwibuild.^Container$.0: 'description_scm' is a required property Failed validation in kiwibuild.^Container$.0: 'description_path' is a required property Failed validation in kiwibuild.^Container$.1: 'description_scm' is a required property Failed validation in kiwibuild.^Container$.1: 'description_path' is a required property Failed validation in kiwibuild.^Container$.2: 'description_scm' is a required property Failed validation in kiwibuild.^Container$.2: 'description_path' is a required property WARNING: Unrecognized config option: global_description_scm. Did you mean global_version? WARNING: Unrecognized config option: global_description_path. ``` ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
kevin commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` ok, that fixed that. Lets gooooo...
``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
kevin merged a pull-request against the project: `pungi-fedora` that you are following.
Merged pull-request:
`` Switch Fedora Cloud and Container builds to use kiwibuild ``
ngompa commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` :tada: ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
adamwill commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` okay, so good news, we got a compose, bad news, all the kiwibuilds failed. This means we have *two* problems: one, all the kiwibuilds are failing, two, the compose isn't dying when that happens (like it should).
The kiwibuilds are dying because their "package names" (remember, everything's a package so far as Koji is concerned) aren't listed somewhere they should be: "BuildError: package (image) Fedora-Cloud-Base-AmazonEC2 not in list for tag f41-updates-candidate" - see https://koji.fedoraproject.org/koji/taskinfo?taskID=114675502 . I presume we have some mechanism somewhere for adding these notional "package names" to the appropriate tags, I'll have to find out the 'package name' for imagefactory builds and grep around for it.
The compose isn't failing (as it should, because some of the images aren't supposed to be "failable") because the kiwibuild phase of pungi doesn't handle "failability" right. I'll file a Pungi issue on that. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
adamwill commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` https://pagure.io/pungi/pull-request/1738 should fix the failability issue, I hope. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
adamwill commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` So the "package name" for ImageFactory cloud base images is "Fedora-Cloud-Base", but I can't yet find anywhere in an obvious repo where we script adding that package name to Koji tags. Is this maybe just done on an ad hoc basis and we need @kevin or someone to do it for all the Kiwi "package names"? I think the list is:
* Fedora-Cloud-Base-Generic * Fedora-Cloud-Base-AmazonEC2 * Fedora-Cloud-Base-GCE * Fedora-Cloud-Base-Vagrant-libvirt * Fedora-Cloud-Base-Vagrant-VirtualBox * Fedora-Container-Base-Generic * Fedora-Container-Base-Generic-Minimal
Or we could, I guess, find out what names ImageFactory used for all of those and get Kiwi to use the same ones.
There *are* two cases where Kiwi used the same name as ImageFactory - Fedora-Container-Toolbox and Fedora-Cloud-Base-Azure . In those cases, there was yet a different failure: we tried to do `dnf5 group install kiwi-build` and it failed.
DEBUG package_manager.py:289: ['/usr/bin/dnf5', '--installroot', '/var/lib/mock/f41-build-49556467-5901287/root/', 'group', 'install', 'kiwi-build', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes'] DEBUG util.py:636: child environment: None DEBUG util.py:553: Using nspawn with args ['--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.aqfgvrhs:/etc/resolv.conf'] DEBUG util.py:558: Executing command: ['/usr/bin/systemd-nspawn', '-q', '-M', '5df3ab93122943c1ae84aebc3e29ec84', '-D', '/var/lib/mock/f41-build-49556467-5901287-bootstrap/root', '-a', '--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.aqfgvrhs:/etc/resolv.conf', '--console=pipe', '--setenv=TERM=vt100', '--setenv=SHELL=/bin/bash', '--setenv=HOME=/var/lib/mock/f41-build-49556467-5901287/root/installation-homedir', '--setenv=HOSTNAME=mock', '--setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin', '--setenv=PROMPT_COMMAND=printf "\033]0;<mock-chroot>\007"', '--setenv=PS1=<mock-chroot> \s-\v\$ ', '--setenv=LANG=C.UTF-8', '--setenv=LC_MESSAGES=C.UTF-8', '--resolv-conf=off', '/usr/bin/dnf5', '--installroot', '/var/lib/mock/f41-build-49556467-5901287/root/', 'group', 'install', 'kiwi-build', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes', '--setopt=tsflags=nocontexts'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/var/lib/mock/f41-build-49556467-5901287/root/installation-homedir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\033]0;<mock-chroot>\007"', 'PS1': '<mock-chroot> \s-\v\$ ', 'LANG': 'C.UTF-8', 'LC_MESSAGES': 'C.UTF-8', 'SYSTEMD_NSPAWN_TMPFS_TMP': '0', 'SYSTEMD_SECCOMP': '0'} and shell False DEBUG util.py:463: Updating and loading repositories: DEBUG util.py:463: build 100% | 73.0 MiB/s | 17.2 MiB | 00m00s DEBUG util.py:463: Repositories loaded. DEBUG util.py:461: Failed to resolve the transaction: DEBUG util.py:461: No match for argument: kiwi-build DEBUG util.py:610: Child return code was: 1
looks like the 'build' repo we are using does not have that package group in it. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
adamwill commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` Kiwi ImageFactory
Fedora-Cloud-Base-Generic Fedora-Cloud-Base Fedora-Cloud-Base-AmazonEC2 Fedora-Cloud-Base [0] Fedora-Cloud-Base-GCE Fedora-Cloud-Base-GCP Fedora-Cloud-Base-Vagrant-libvirt Fedora-Cloud-Base-Vagrant Fedora-Cloud-Base-Vagrant-VirtualBox Fedora-Cloud-Base-Vagrant [1] Fedora-Cloud-Base-Azure N/A [2] Fedora-Container-Base-Generic Fedora-Container-Base Fedora-Container-Base-Generic-Minimal Fedora-Container-Minimal-Base
so we can't reuse the ImageFactory names perfectly, but we could get closer if we wanted to...
[0] The ImageFactory Fedora-Cloud-Base task builds both .qcow2 and .raw.xz images. For Kiwi, Fedora-Cloud-Base-Generic builds .qcow2, Fedora-Cloud-Base-AmazonEC2 builds .raw.xz (I think) [1] The ImageFactory Fedora-Cloud-Base-Vagrant task builds both libvirt and VirtualBox images. [2] We don't build an Azure image with ImageFactory. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
adamwill commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` sorry, I was wrong about `Fedora-Cloud-Base-Azure` in the match list above, it should be the same name for ImageFactory, as I said in the previous comment. Can't edit comments on this repo for some reason. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
kevin commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` I'm fine just adding the new names... unless that will cause problems anywhere looking for/using them?
``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
ngompa commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` I'd prefer to add the new names if we can. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
kevin commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` Yeah, just done that. We can adjust later if there's any problems doing that...
``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
adamwill commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` So I *think* we're finally done cleaning up all the consequences of this. Well. It might be good to compare package manifests before and after the switch for each image, I guess. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
adamwill commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` @jcline did that for container_base at least (thanks jcline!) and the results are at https://pagure.io/fedora-kiwi-descriptions/pull-request/29 . We should do the same for container base minimal, toolbox, and cloud images, I guess. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
jcline commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: ``
@jcline did that for container_base at least (thanks jcline!) and the results are at https://pagure.io/fedora-kiwi-descriptions/pull-request/29 . We should do the same for container base minimal, toolbox, and cloud images, I guess.
That should also cover the minimal image, and I can take a look at the toolbox and cloud images next. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
adamwill commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` were you able to do those comparisons, @jcline ? ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
jcline commented on the pull-request: `Switch Fedora Cloud and Container builds to use kiwibuild` that you are following: `` Ah, whoops, forgot to update. The toolbox looked very similar, a few unnecessary libraries went away. I haven’t looked at the Amazon, Google, or vagrant box yet. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1248
rel-eng@lists.fedoraproject.org
-
Adam Williamson -
Jeremy Cline
-
Kevin Fenzi
-
Neal Gompa