supakeen opened a new pull-request against the project: `pungi-fedora` that you are following: `` workstation: build with osbuild ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
kevin commented on the pull-request: `workstation: build with osbuild` that you are following: `` I'm not sure how this will interact with the image names/copying part of things. I guess since it has another name it should be fine.
Perhaps @lsedlar could take a quick glance? ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `workstation: build with osbuild` that you are following: `` we usually capitalize the elements of the name, I think - e.g. Fedora-Workstation-Live , Fedora-Cloud-Base-Vagrant . It's not really important, but it'd just be a bit inconsistent for this to be lower case.
What I'm concerned about is what the productmd properties of the image wind up as. If it has the exact same subvariant, type and format as the 'real' Workstation live that's an issue, because that's the triplet of identifying features that fedfind and all the things built off it (mainly my openQA industrial complex, but also a few other things I think) use to 'identify' images.
If the "image type" that's written into the productmd metadata is instead "live-installer" (as this change *seems like* it might result in), that causes a different problem: that is not a valid image type according to productmd - https://github.com/release-engineering/productmd/blob/master/productmd/image... .
But in fact, looking at the pungi code, it almost seems like the image type in the productmd metadata might *actually* be set to "dvd" - I think that's what https://pagure.io/pungi/blob/master/f/pungi/phases/osbuild.py#_249 does - which is weird in a *third* way, because the "dvd" image type is meant to indicate a traditional DVD installer image, like the Fedora Server DVD.
I think the way I'd ideally *want* this to work is that we add a new image type to productmd, "live-osbuild" or "osbuild-live" or something like that, and have that be the type used in the productmd metadata for these images. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
supakeen commented on the pull-request: `workstation: build with osbuild` that you are following: `` @adamwill and @kevin, thanks for your comments. I've addressed the `name` inconsistency and I'll dive into what productmd means. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
lsedlar commented on the pull-request: `osbuild: build workstation` that you are following: `` What image format and type Pungi writes into the metadata ultimately depends on what is created in Koji. The "archive" uploaded to Koji will have some `type_name`. That value is used in metadata as `type` (except `iso` is changed to `dvd`).
Format is set to a key from [this dict](https://pagure.io/pungi/blob/master/f/pungi/phases/image_build.py#_26) which contains the type name. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` @lsedlar yeah, the bit where I get a bit lost is what the `type_name` for these will be. My eyes started glazing over right around the point of trying to figure that out :D ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` supakeen: productmd - https://github.com/release-engineering/productmd - is the project that defines the metadata format we're concerned with here - the format used by the files in the `metadata/` directory of any compose, e.g. https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230728.n... - and provides some tools and parsers for working with it. pungi should never be set up to write things into the metadata that aren't within the scope defined in the productmd project. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
lsedlar commented on the pull-request: `osbuild: build workstation` that you are following: `` The `type_name` is not driven by Pungi. It's set by OSBuild when it uploads the build into Koji. For example [Fedora-IoT-ostree](https://koji.fedoraproject.org/koji/buildinfo?buildID=2265689) has it set to `iso`. I'm pretty sure it's going to be `iso` for the Workstation image too. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` well, that's a problem, because that's not what that type is meant for, as I said. that type is intended to signify a traditional DVD installer - the Server DVD, or the old Everything DVD we no longer make.
IoT composes do look weird, now I look at them, because they have these two images:
{ "arch": "x86_64", "bootable": true, "checksums": { "sha256": "6c2c96fa9430e20c93cb600660f35461cf17ba23f2c7da30e5a7c902459fb18c" }, "disc_count": 1, "disc_number": 1, "format": "iso", "implant_md5": "cc67ff36b9613545c4dc58355a8ac5ef", "mtime": 1690895814, "path": "IoT/x86_64/iso/Fedora-IoT-ostree-x86_64-39-20230801.0.iso", "size": 1336844288, "subvariant": "IoT", "type": "dvd-ostree", "volume_id": "Fedora-iot-ostree-x86_64-39" }
and
{ "arch": "x86_64", "bootable": false, "checksums": { "sha256": "5a002c2bdb0eb7269ab507eacc272c99d3142a74708aa0565612d787827da44f" }, "disc_count": 1, "disc_number": 1, "format": "iso", "implant_md5": null, "mtime": 1690896711, "path": "IoT/x86_64/iso/Fedora-IoT-ostree-39.20230801.0-20230801.0.x86_64.iso", "size": 2411200512, "subvariant": "IoT", "type": "dvd", "volume_id": null },
the first is the 'normal' IoT installer image, note the type `dvd-ostree`, which is the correct type for this kind of image. Are you saying the second is a similar image but built by osbuild? If so, that's not really what its metadata is identifying it as, which is kind of an issue. We probably need to work this out so we have something consistent and usable. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
supakeen commented on the pull-request: `osbuild: build workstation` that you are following: `` Ok, spent the morning reading around. @lsedlar I'm a bit confused on the:
The type_name is not driven by Pungi. It's set by OSBuild when it uploads the build into Koji.
Bit. It seems we set the type_name to image and koji then auto detects based on the file extensions what type it is. This is then received by pungi which translates (in the osbuild-case) iso to dvd and otherwise lets it set to what koji auto detected.
My plan would be to include an archive_type name in the osbuild pungi config to override what pungi writes into the metadata, does this sounds like a good approach? (I'd also fix the dvd vs dvd-ostree thing). ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` In terms of what the metadata for these images should actually *be*, I don't think they should have identical subvariant, type and format to the images built with a different tool, as that's confusing. In fact, now I recall, the productmd docs [specifically say you can't do that](https://github.com/release-engineering/productmd/blob/master/doc/images-1.1....):
"It is required that the combination of subvariant, type, format, arch and disc_number attributes is unique to each image in the compose."
...and there is a [function that identifies images according to those properties](https://github.com/release-engineering/productmd/blob/master/productmd/image...). Also, there's [a check](https://github.com/release-engineering/productmd/blob/master/productmd/image...) which I think might cause composes to blow up if you actually *do* manage to cause an image to have the same identifying properties as an existing image (assuming the compose process runs through that function, which I think it does).
As the image types are [defined in productmd](https://github.com/release-engineering/productmd/blob/master/productmd/image...), we can't just make up new ones for these images, and we shouldn't really 'abuse' existing types to mean something other than they're meant to (as we're currently doing here). Probably the easiest thing to do would be to give these images different subvariants, however we need to express that in Pungi config (or whatever). So the IoT installer image built by osbuild could be subvariant "IoT-osbuild", type "dvd-ostree". ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
supakeen commented on the pull-request: `osbuild: build workstation` that you are following: `` Perhaps it would be cleaner to add a `built-by` to the unique tuple that contains what pungi initially used to build the compose? ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` that would be cleaner, but a much more major change and would require changes to quite a lot of stuff downstream that I'm aware of (and probably more that I'm not).
In openQA we use just subvariant, type, format and arch and expect that combination to be unique, as we know Fedora has no multi-disc deliverables ATM. So for instance all the 'flavors' in openQA are subvariant-type-format combinations:
`Workstation-live-iso` `Silverblue-dvd_ostree-iso`
etc. We schedule tests by basically checking the productmd data for each image, seeing if it matches one of those flavor names, and if it does, we schedule those tests on that image. This is all going to break if we have *two* images that have the same subvariant, type and format (I'm not sure exactly what would happen, but either the scheduler would blow up or we'd just randomly schedule the tests on one image or the other each time without really knowing which it was), and changing this to also include a 'builtby' property would require changes to the scheduler, the openQA test config, the code for reporting results to the wiki, and probably various other places I haven't remembered yet, since the flavor names have been baked in for a long time. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` I could of course stick ugly special-case hacks in the scheduler to make it 'know' which of the two otherwise-identical images for some osbuild case to use, and keep the flavor names the same, but I'd really like to *not* have to do that... ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
supakeen commented on the pull-request: `osbuild: build workstation` that you are following: `` I've created a PR in the productmd repository to add a live-osbuild image type so we can produce under that. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` That's an option, sure. You should probably also add a `dvd-ostree-osbuild` type for the IoT image, in that case. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` Would the idea here be that we only use these new types while we're transitioning, then when we drop the 'old' way and make the osbuild images the only ones, we switch them to use the original types? ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
supakeen commented on the pull-request: `osbuild: build workstation` that you are following: `` Updated the PR in productmd to also include `dvd-ostree-osbuild`. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
supakeen commented on the pull-request: `osbuild: build workstation` that you are following: ``
Would the idea here be that we only use these new types while we're transitioning, then when we drop the 'old' way and make the osbuild images the only ones, we switch them to use the original types?
That would be my idea for it yes, initially publish them under the different image types until found to be good then we'd only need to switch the image types? ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
supakeen commented on the pull-request: `osbuild: build workstation` that you are following: `` The productmd PR got merged so I'll be updating this PR with the new image types added there. Does productmd need to get deployed somewhere before it will recognize the new types? ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
supakeen commented on the pull-request: `osbuild: build workstation` that you are following: `` I've created (another) PR to let the osbuild phase in pungi take its manifest type (as used by productmd) from the pungi configuration instead of the koji output: https://pagure.io/pungi/pull-request/1700 ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` Yeah, I *think* we'd need to deploy productmd with the fixes on whatever machine controls the composes (not the machines that run the individual image builds, but wherever the whole compose process is run from). @kevin and/or @humaton should be able to help with that. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
kevin commented on the pull-request: `osbuild: build workstation` that you are following: `` Yeah, that would be compose-rawhide01 for rawhide composes.
We will need a f38 update of the new productmd package. I can do a infra specific one if there's some reason to not just do a real f38 one. Let me know... ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` I think doing a real f38 build would be fine, I just wanted your expertise on what release we needed a build for and where it needed to go :) ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` Looks like lsedlar already tagged a release, did builds, and updates. My kinda maintainer! https://bodhi.fedoraproject.org/updates/FEDORA-2023-b0fc79a875 is the F38 one. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
supakeen commented on the pull-request: `osbuild: build workstation` that you are following: `` @adamwill Yes, and @lsedlar also accepted the PR I created in pungi this morning and submitted updates. I'll be updating this PR to use the newly added `manifest_type` in the configuration which sets the correct value for productmd.
I think after the above updates have moved through bodhi and this PR is updated we'd be ready to merge :) ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
kevin commented on the pull-request: `osbuild: build workstation` that you are following: `` sounds good. I updated compose-rawhide01 to the new productmd package. let me know if anything more is needed. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` We would need the compose to use pungi-4.5.0-2 - [here's the F38 update](https://bodhi.fedoraproject.org/updates/FEDORA-2023-7189241e33). If that's in place, I think we could go ahead and merge this PR and give it a shot. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
supakeen commented on the pull-request: `osbuild: build workstation` that you are following: ``
We would need the compose to use pungi-4.5.0-2 - [here's the F38 update](https://bodhi.fedoraproject.org/updates/FEDORA-2023-7189241e33). If that's in place, I think we could go ahead and merge this PR and give it a shot.
Yep, just updated this PR to actually set `manifest_type` so it should be ready after pungi is deployed. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
kevin commented on the pull-request: `osbuild: build workstation` that you are following: `` All the composers are now using pungi-4.5.0-2.
Anything else? or shall we merge this finally and see what happens? ;) ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
supakeen commented on the pull-request: `osbuild: build workstation` that you are following: ``
All the composers are now using pungi-4.5.0-2.
Anything else? or shall we merge this finally and see what happens? ;)
Let's merge and see what happens! ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
kevin commented on the pull-request: `osbuild: build workstation` that you are following: `` Alright. :) ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
kevin merged a pull-request against the project: `pungi-fedora` that you are following.
Merged pull-request:
`` osbuild: build workstation ``
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` ahh, so this was the giant explosion I saw on my way home from badminton...:D ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
kevin commented on the pull-request: `osbuild: build workstation` that you are following: `` small bit o doom:
Failed validation in osbuild.^Workstation$.0.repo: 'Everythin changed that to an array and refired. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
supakeen commented on the pull-request: `osbuild: build workstation` that you are following: ``
small bit o doom:
Failed validation in osbuild.^Workstation$.0.repo: 'Everythin changed that to an array and refired.
Sounds like tiny doom indeed, I can make a separate PR if everything is working after that change?
Is there a way for me to validate that it's going to work before I make that PR? ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
kevin commented on the pull-request: `osbuild: build workstation` that you are following: `` I just went and pushed that change already directly yesterday. ;)
So, it looks like it fired it off, but it failed:
https://koji.fedoraproject.org/koji/taskinfo?taskID=106979354
``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` so that seems to be because of the fact that both sdubby and grubby are in `@ancaonda-tools` in comps. They are known to conflict. This is why sdubby is [excluded in `fedora-live-base.ks`](https://pagure.io/fedora-kickstarts/blob/main/f/fedora-live-base.ks#_42). So I guess the question is, why does that exclusion not take effect here? ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` It's notable that, compared to the logs we get from the current live image workflow, I can't find any record of the kickstart that was used. The current live image workflow includes the actual flattened kickstart as one of the Koji task outputs. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
supakeen commented on the pull-request: `osbuild: build workstation` that you are following: `` @adamwill `osbuild-composer` does not use kickstart files to define distributions (and editions/spins). In this initial version it is defined as Go code, Go code that wasn't updated with the change to exclude sdubby (I took over the kickstart packages section when I first created support for the live-installer).
`osbuild-composer` defines customizations (such as packages) in a toml format called 'blueprints' which will take over the role of defining Fedora's editions and spins but that's not *currently* the case for this image. When that switch is made, which is planned for this quarter I believe you'd be receiving the blueprint used for the build and the generated manifest in the included files in the output. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
ngompa commented on the pull-request: `osbuild: build workstation` that you are following: `` The plan is to allow osbuild to have blueprint-driven image builds, right? We can't have stuff happening in the Go code that nobody can fix in time because we are relying on Red Hat infrastructure (not even Fedora infrastructure!). ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
ngompa commented on the pull-request: `osbuild: build workstation` that you are following: `` (By blueprint-driven, I mean everything defined and triggered in blueprints, rather than relying on Go code for detecting OSes, stages, and configuration steps) ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
supakeen commented on the pull-request: `osbuild: build workstation` that you are following: `` @ngompa That is correct. I'd like to have a repository much like fedora-kickstarts which would contain blueprints instead. For a start I'm implementing three editions (Workstation, KDE, Server) as blueprints to find out what we're lacking and where any pain points are with blueprints. ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
adamwill commented on the pull-request: `osbuild: build workstation` that you are following: `` So I wound up back in this area after noticing F40 IoT installer images aren't being tested in openQA because they're built by osbuild and their type is set to 'dvd', the same problem we had here.
Since this seems like it'll keep cropping up, I sent [a PR to improve the pungi autodetection](https://pagure.io/pungi/pull-request/1707#)... ``
To reply, visit the link below or just reply to this email https://pagure.io/pungi-fedora/pull-request/1190
rel-eng@lists.fedoraproject.org
-
Adam Williamson -
Kevin Fenzi
-
Lubomír Sedlář
-
Neal Gompa
-
Simon de Vlieger