Hello.
I'm from the Image Builder team at Red Hat. For those who don't know it, our team works on tooling for building various OS images and integrates with Koji as a CG (https://github.com/osbuild/koji-osbuild).
Our current integration works in the following way: 1. Koji builder plugin makes a REST API call to our service. 2. Our service builds the image, uploads the image file to Koji and then imports it using CG API ( https://github.com/osbuild/osbuild-composer/blob/eb8b63f3ca166ecad13e10a21b8... ). 3. Koji builder plugin waits for the image build to finish and then downloads logs and osbuild manifest (a kind of recipe allowing to rebuild the image) and uploads them to the build task.
As a result, valuable files (such as logs and the osbuild manifest) are not attached to the build in Koji, but to the build task. Files attached to the task usually get garbage collected after some time (I guess depending on the Koji instance configuration), so as a result we are losing these files after some time.
I'm currently reworking the way osbuild-composer imports content to Koji, so that we attach build logs and osbuild manifests to the Koji build as well. When we import outputs to Koji, we need to specify the output build type. For the image, we use "image", for logs we'll use "log", but the osbuild manifest would probably benefit from its own build type (e.g. "osbuild-manifest").
From what I checked, some content generators use their own build type (e.g. when I run "koji --noauth call listBTypes" with Fedora Koji, there's a "module" build type for MBS).
I would like to ask what is the common or usual way for a Content Generator to mandate that a custom build type is available in Koji when using it? My understanding is that a new build type can be added to the Koji instance by an admin running the appropriate command. Is there some other way?
I'm trying to determine what's the best way to handle the situation that our CG will start using a custom build type, while being already used by multiple Koji instances. Is the only way to document this requirement and work with responsible admins of Koji instances, which integrate with our CG, to adjust the configuration before we start using the new build type?
Thanks in advance for any guidance.
Regards, Tomas
*"There are only two hard things in computer science: cache invalidation and naming things."*
Unfortunately, there are a number of different "types" in Koji, and it looks like you may be mixing some of them up.
"btypes" are Build Types. This is primarily a way to indicate the type of a build, not individual outputs. It gets confusing because a build can have multiple btypes, and the output archives that are part of the build can have metadata for the btype.
Koji also has archivetypes, which are types for the archives themselves rather than the build as a whole. Unfortunately, these are really simplistic and really only serve to limit which file extensions can be imported.
And just to make things more confusing, the content generator metadata has a type field for outputs that is neither of these. The output type can be rpm, file, or log.
In general, I would discourage creating builds with multiple btypes.
Logs are a special case in Koji. They are neither rpms nor archives and they do not have a db entry. They are simply stored on disk.You don't need a btype to attach logs.
Lastly, to answer the remaining part of your question. There is no mechanism for a content generator to define btypes in the db. As you surmised, this must be managed by an admin. I would rather avoid having a vast proliferation of btypes.
On Thu, Sep 7, 2023 at 9:34 AM Tomáš Hozza thozza@redhat.com wrote:
Hello.
I'm from the Image Builder team at Red Hat. For those who don't know it, our team works on tooling for building various OS images and integrates with Koji as a CG (https://github.com/osbuild/koji-osbuild).
Our current integration works in the following way:
- Koji builder plugin makes a REST API call to our service.
- Our service builds the image, uploads the image file to Koji and then
imports it using CG API ( https://github.com/osbuild/osbuild-composer/blob/eb8b63f3ca166ecad13e10a21b8... ). 3. Koji builder plugin waits for the image build to finish and then downloads logs and osbuild manifest (a kind of recipe allowing to rebuild the image) and uploads them to the build task.
As a result, valuable files (such as logs and the osbuild manifest) are not attached to the build in Koji, but to the build task. Files attached to the task usually get garbage collected after some time (I guess depending on the Koji instance configuration), so as a result we are losing these files after some time.
I'm currently reworking the way osbuild-composer imports content to Koji, so that we attach build logs and osbuild manifests to the Koji build as well. When we import outputs to Koji, we need to specify the output build type. For the image, we use "image", for logs we'll use "log", but the osbuild manifest would probably benefit from its own build type (e.g. "osbuild-manifest").
From what I checked, some content generators use their own build type (e.g. when I run "koji --noauth call listBTypes" with Fedora Koji, there's a "module" build type for MBS).
I would like to ask what is the common or usual way for a Content Generator to mandate that a custom build type is available in Koji when using it? My understanding is that a new build type can be added to the Koji instance by an admin running the appropriate command. Is there some other way?
I'm trying to determine what's the best way to handle the situation that our CG will start using a custom build type, while being already used by multiple Koji instances. Is the only way to document this requirement and work with responsible admins of Koji instances, which integrate with our CG, to adjust the configuration before we start using the new build type?
Thanks in advance for any guidance.
Regards, Tomas -- Tomáš Hozza Image Builder, Linux Engineering
PGP: 908AF740C0F898A1079968CDC5887AD51D9F3C2D UTC+2 (CEST) / UTC+1 (CET) Red Hat Czech http://cz.redhat.com _______________________________________________ koji-devel mailing list -- koji-devel@lists.fedorahosted.org To unsubscribe send an email to koji-devel-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahosted.o... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Thanks for the reply.
I'm replying using the hyperkitty UI, since I didn't get your response via email, so hopefully this will work...
*"There are only two hard things in computer science: cache invalidation and naming things."*
Unfortunately, there are a number of different "types" in Koji, and it looks like you may be mixing some of them up.
"btypes" are Build Types. This is primarily a way to indicate the type of a build, not individual outputs. It gets confusing because a build can have multiple btypes, and the output archives that are part of the build can have metadata for the btype.
Koji also has archivetypes, which are types for the archives themselves rather than the build as a whole. Unfortunately, these are really simplistic and really only serve to limit which file extensions can be imported.
And just to make things more confusing, the content generator metadata has a type field for outputs that is neither of these. The output type can be rpm, file, or log.
Right. I feel sufficiently confused already :)
In CG_Importer->prep_outputs(), if the output "type" is not "rpm" or "log", the CG_Importer->prep_archive() is called (https://pagure.io/koji/blob/master/f/kojihub/kojihub.py#_7047).
In that method, the output "type" does not play any role and the code determines the output "btype" (which I read as "build type") from the extra metadata attached to the output. It must be set under the "typeinfo" key of the extra metadata dict (if not a legacy type) and it must be a supported build type of the Koji instance. If it is not set at all, the import will fail (https://pagure.io/koji/blob/master/f/kojihub/kojihub.py#_7091). It must be also listed in the build extra metadata.
That's the reason why I'm asking about build types in Koji. While there are various types of types, from what I can tell it always boils down to "build type" when importing outputs to Koji as a CG... Each output must have a valid build type set under "typeinfo" key in its extra metadata if it is not of type "log" or "rpm".
In general, I would discourage creating builds with multiple btypes.
Let me rephrase my question:
In order enable reproducible builds of images, image builder as the CG needs to attach osbuild manifests (basically JSON files) to a Koji build, along with images themselves. I don't think that it makes sense to mark these manifests as "image", since they are not images. But they are also not "log". Is there any other sensible way to attach such additional files to a Koji build, other than defining a new custom "build type" or marking them as an "image" or a "log"?
While inspecting kojihub's code, I didn't find any, but I wanted to check with you in case there is some common approach to this problem.
Logs are a special case in Koji. They are neither rpms nor archives and they do not have a db entry. They are simply stored on disk.You don't need a btype to attach logs.
Lastly, to answer the remaining part of your question. There is no mechanism for a content generator to define btypes in the db. As you surmised, this must be managed by an admin. I would rather avoid having a vast proliferation of btypes.
I agree that this does not scale, but AFAICT there is no "generic file" build type that a CG could use to import non-standard files as part of a Koji build. One must choose from existing "built-in" or custom build types or mark the file as a log.
Regards, Tomas
The data model is maybe not the best here, as it evolved from what was originally a system for just rpms and then expanded organically over time, but it is what it is.
The main thing here is that the btype is the type of /the build/, not the type of any particular component (it stands for build type after all). From what I gather, the builds you are producing are, fundamentally, images. The manifests you are concerned about are clearly supplementary data related to said image build. So to me, it makes sense to include them as artifacts of the image build entry. I do not see a reason to create a separate build type for them, any more than I would create a separate build type for srpms or logs.
That said, there is clearly prior art where plugins have created btypes for similar supplementary (operator-manifests, remote-source-file, remote-sources, icm). Had I been asked when these were added, I likely would have advised against them as well.
So, let's say you take my advice and just use the image btype. When you put the 'image' entry under `typeinfo` for your manifest, you are not saying this file is an image file, you are saying it is part of an image build. Whatever data you put under typeinfo.image is arshive-specific metadata for the image build.
OTOH, you could go the route of adding another btype. If so, you would likely want to document that admins need to add this btype if they want to use your plugin.
On Mon, Sep 11, 2023 at 11:23 AM Tomas Hozza thozza@redhat.com wrote:
Thanks for the reply.
I'm replying using the hyperkitty UI, since I didn't get your response via email, so hopefully this will work...
*"There are only two hard things in computer science: cache invalidation and naming things."*
Unfortunately, there are a number of different "types" in Koji, and it looks like you may be mixing some of them up.
"btypes" are Build Types. This is primarily a way to indicate the type
of a
build, not individual outputs. It gets confusing because a build can have multiple btypes, and the output archives that are part of the build can have metadata for the btype.
Koji also has archivetypes, which are types for the archives themselves rather than the build as a whole. Unfortunately, these are really simplistic and really only serve to limit which file extensions can be imported.
And just to make things more confusing, the content generator metadata
has
a type field for outputs that is neither of these. The output type can be rpm, file, or log.
Right. I feel sufficiently confused already :)
In CG_Importer->prep_outputs(), if the output "type" is not "rpm" or "log", the CG_Importer->prep_archive() is called ( https://pagure.io/koji/blob/master/f/kojihub/kojihub.py#_7047).
In that method, the output "type" does not play any role and the code determines the output "btype" (which I read as "build type") from the extra metadata attached to the output. It must be set under the "typeinfo" key of the extra metadata dict (if not a legacy type) and it must be a supported build type of the Koji instance. If it is not set at all, the import will fail (https://pagure.io/koji/blob/master/f/kojihub/kojihub.py#_7091). It must be also listed in the build extra metadata.
That's the reason why I'm asking about build types in Koji. While there are various types of types, from what I can tell it always boils down to "build type" when importing outputs to Koji as a CG... Each output must have a valid build type set under "typeinfo" key in its extra metadata if it is not of type "log" or "rpm".
In general, I would discourage creating builds with multiple btypes.
Let me rephrase my question:
In order enable reproducible builds of images, image builder as the CG needs to attach osbuild manifests (basically JSON files) to a Koji build, along with images themselves. I don't think that it makes sense to mark these manifests as "image", since they are not images. But they are also not "log". Is there any other sensible way to attach such additional files to a Koji build, other than defining a new custom "build type" or marking them as an "image" or a "log"?
While inspecting kojihub's code, I didn't find any, but I wanted to check with you in case there is some common approach to this problem.
Logs are a special case in Koji. They are neither rpms nor archives and they do not have a db entry. They are simply stored on disk.You don't
need
a btype to attach logs.
Lastly, to answer the remaining part of your question. There is no mechanism for a content generator to define btypes in the db. As you surmised, this must be managed by an admin. I would rather avoid having a vast proliferation of btypes.
I agree that this does not scale, but AFAICT there is no "generic file" build type that a CG could use to import non-standard files as part of a Koji build. One must choose from existing "built-in" or custom build types or mark the file as a log.
Regards, Tomas _______________________________________________ koji-devel mailing list -- koji-devel@lists.fedorahosted.org To unsubscribe send an email to koji-devel-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahosted.o... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Thanks for your reply.
After this discussion and after seeing that e.g. Image Factory also imports kickstarts and some XML files into the build as "image" btype, I'm convinced that this is a reasonable approach also for osbuild manifests.
Regards, Tomas
On Mon, Sep 11, 2023 at 9:49 PM Michael McLean mikem@redhat.com wrote:
The data model is maybe not the best here, as it evolved from what was originally a system for just rpms and then expanded organically over time, but it is what it is.
The main thing here is that the btype is the type of /the build/, not the type of any particular component (it stands for build type after all). From what I gather, the builds you are producing are, fundamentally, images. The manifests you are concerned about are clearly supplementary data related to said image build. So to me, it makes sense to include them as artifacts of the image build entry. I do not see a reason to create a separate build type for them, any more than I would create a separate build type for srpms or logs.
That said, there is clearly prior art where plugins have created btypes for similar supplementary (operator-manifests, remote-source-file, remote-sources, icm). Had I been asked when these were added, I likely would have advised against them as well.
So, let's say you take my advice and just use the image btype. When you put the 'image' entry under `typeinfo` for your manifest, you are not saying this file is an image file, you are saying it is part of an image build. Whatever data you put under typeinfo.image is arshive-specific metadata for the image build.
OTOH, you could go the route of adding another btype. If so, you would likely want to document that admins need to add this btype if they want to use your plugin.
On Mon, Sep 11, 2023 at 11:23 AM Tomas Hozza thozza@redhat.com wrote:
Thanks for the reply.
I'm replying using the hyperkitty UI, since I didn't get your response via email, so hopefully this will work...
*"There are only two hard things in computer science: cache invalidation and naming things."*
Unfortunately, there are a number of different "types" in Koji, and it looks like you may be mixing some of them up.
"btypes" are Build Types. This is primarily a way to indicate the type
of a
build, not individual outputs. It gets confusing because a build can
have
multiple btypes, and the output archives that are part of the build can have metadata for the btype.
Koji also has archivetypes, which are types for the archives themselves rather than the build as a whole. Unfortunately, these are really simplistic and really only serve to limit which file extensions can be imported.
And just to make things more confusing, the content generator metadata
has
a type field for outputs that is neither of these. The output type can
be
rpm, file, or log.
Right. I feel sufficiently confused already :)
In CG_Importer->prep_outputs(), if the output "type" is not "rpm" or "log", the CG_Importer->prep_archive() is called ( https://pagure.io/koji/blob/master/f/kojihub/kojihub.py#_7047).
In that method, the output "type" does not play any role and the code determines the output "btype" (which I read as "build type") from the extra metadata attached to the output. It must be set under the "typeinfo" key of the extra metadata dict (if not a legacy type) and it must be a supported build type of the Koji instance. If it is not set at all, the import will fail (https://pagure.io/koji/blob/master/f/kojihub/kojihub.py#_7091). It must be also listed in the build extra metadata.
That's the reason why I'm asking about build types in Koji. While there are various types of types, from what I can tell it always boils down to "build type" when importing outputs to Koji as a CG... Each output must have a valid build type set under "typeinfo" key in its extra metadata if it is not of type "log" or "rpm".
In general, I would discourage creating builds with multiple btypes.
Let me rephrase my question:
In order enable reproducible builds of images, image builder as the CG needs to attach osbuild manifests (basically JSON files) to a Koji build, along with images themselves. I don't think that it makes sense to mark these manifests as "image", since they are not images. But they are also not "log". Is there any other sensible way to attach such additional files to a Koji build, other than defining a new custom "build type" or marking them as an "image" or a "log"?
While inspecting kojihub's code, I didn't find any, but I wanted to check with you in case there is some common approach to this problem.
Logs are a special case in Koji. They are neither rpms nor archives and they do not have a db entry. They are simply stored on disk.You don't
need
a btype to attach logs.
Lastly, to answer the remaining part of your question. There is no mechanism for a content generator to define btypes in the db. As you surmised, this must be managed by an admin. I would rather avoid having
a
vast proliferation of btypes.
I agree that this does not scale, but AFAICT there is no "generic file" build type that a CG could use to import non-standard files as part of a Koji build. One must choose from existing "built-in" or custom build types or mark the file as a log.
Regards, Tomas _______________________________________________ koji-devel mailing list -- koji-devel@lists.fedorahosted.org To unsubscribe send an email to koji-devel-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahosted.o... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
koji-devel@lists.fedorahosted.org
-
Michael McLean -
Tomas Hozza -
Tomáš Hozza