On Mon, Apr 24, 2017 at 05:06:47PM +0200, Stanislav Ochotnicky wrote:
On Mon 24 Apr 2017 04:56:05 PM CEST Jan Kaluža jkaluza@redhat.com wrote:
On 04/24/2017 04:20 PM, Stanislav Ochotnicky wrote:
On Mon 24 Apr 2017 03:25:48 PM CEST Ralph Bean rbean@redhat.com wrote:
On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote:
This is the first stab and what the mbs CG metadata output might look like: https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRLivL...
Just to clarify - in the "build" entry here, this is a build of the "nginx module". The "version" (which is 1) and the "release" (which is 2) are in koji terms, not MBS terms.
The mapping between MBS and koji terms should look something like:
MBS -> Koji --------------- name -> name stream -> version version -> releaseThe only kicker I can see at the moment is the datatype of stream(MBS) versus version(Koji). The MBS stream is a string (master, or f26, or whatever). The Koji version... does it have to be an integer?
Well the version even for rpms is a string. Usually semantic versioning style thing - but not necessarily. Of course if you use something lime rpmdev-vercmp on these mbs builds EVRs it might make things weird. I.e.: rpmdev-vercmp master-20 version2-10 master-20 < version2-10
But I think that's OK.
One thing however - can stream contain dash? Because version string cannot. I'd be tempted to just replace it with underscore hoping there won't be both :-)
We currently set the the dist-git branch as a stream value. From MBS point of view, there is no problem with dash in a stream name and I presume there will dashes in a stream name if we don't decide you cannot use dash in dist-git branch name for modules. I would not like to restrict that if there is no real reason to do that.
How much user-visible will the content generator data be? Can for example user search for module in Koji by typing its name-stream-version? In that case, I would start thinking about restricting the branch so it does not contain dash, because people will not replace dash with underscore when searching Koji.
Well this part of content generator is very visible - it's part of the build NVR after all. So things like API calls getBuild(nvr) would be affected by it. And the NVR will need to be input as part of shipping update in Bodhi for example.
I would guess people *will* use dash in the stream name unless they are prevented from doing it?
From discussion in IRC today (paraphrasing others):
We can use a new "build type" for this ("module"). With a new build type (btype) we can:
- store the original MBS name-stream-version in koji as extra information, preserving dashes - while the build has a koji-reformatted name-version-release with dashes converted to underscores.
There is precedent for this in the way Maven GAV is mapped to NVR.
In the CG metadata provided by MBS, btype specific data goes into extra->typeinfo->BTYPENAME for build and archive and this how koji knows the btype. So, the MBS would need to define the mapping from NSV to NVR (hyphens become underscores, etc.) and use the mapped NVR to identify the build. The original, unmodified NSV would be recorded in the "extra" field of the build.
(Archives can only have one btype. builds btypes must match their component archives. btype info can technically be {})
Is that sufficient to solve this mapping problem? Are there any other outstanding questions you need resolved, stano?
(Sorry for this getting splintered across multiple lists...)
On Fri 05 May 2017 07:23:18 PM CEST Ralph Bean rbean@redhat.com wrote:
On Mon, Apr 24, 2017 at 05:06:47PM +0200, Stanislav Ochotnicky wrote:
On Mon 24 Apr 2017 04:56:05 PM CEST Jan Kaluža jkaluza@redhat.com wrote:
On 04/24/2017 04:20 PM, Stanislav Ochotnicky wrote:
On Mon 24 Apr 2017 03:25:48 PM CEST Ralph Bean rbean@redhat.com wrote:
On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote:
This is the first stab and what the mbs CG metadata output might look like: https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRLivL...
Just to clarify - in the "build" entry here, this is a build of the "nginx module". The "version" (which is 1) and the "release" (which is 2) are in koji terms, not MBS terms.
The mapping between MBS and koji terms should look something like:
MBS -> Koji --------------- name -> name stream -> version version -> releaseThe only kicker I can see at the moment is the datatype of stream(MBS) versus version(Koji). The MBS stream is a string (master, or f26, or whatever). The Koji version... does it have to be an integer?
Well the version even for rpms is a string. Usually semantic versioning style thing - but not necessarily. Of course if you use something lime rpmdev-vercmp on these mbs builds EVRs it might make things weird. I.e.: rpmdev-vercmp master-20 version2-10 master-20 < version2-10
But I think that's OK.
One thing however - can stream contain dash? Because version string cannot. I'd be tempted to just replace it with underscore hoping there won't be both :-)
We currently set the the dist-git branch as a stream value. From MBS point of view, there is no problem with dash in a stream name and I presume there will dashes in a stream name if we don't decide you cannot use dash in dist-git branch name for modules. I would not like to restrict that if there is no real reason to do that.
How much user-visible will the content generator data be? Can for example user search for module in Koji by typing its name-stream-version? In that case, I would start thinking about restricting the branch so it does not contain dash, because people will not replace dash with underscore when searching Koji.
Well this part of content generator is very visible - it's part of the build NVR after all. So things like API calls getBuild(nvr) would be affected by it. And the NVR will need to be input as part of shipping update in Bodhi for example.
I would guess people *will* use dash in the stream name unless they are prevented from doing it?
From discussion in IRC today (paraphrasing others):
We can use a new "build type" for this ("module"). With a new build type (btype) we can:
- store the original MBS name-stream-version in koji as extra information, preserving dashes
- while the build has a koji-reformatted name-version-release with dashes converted to underscores.
There is precedent for this in the way Maven GAV is mapped to NVR.
In the CG metadata provided by MBS, btype specific data goes into extra->typeinfo->BTYPENAME for build and archive and this how koji knows the btype. So, the MBS would need to define the mapping from NSV to NVR (hyphens become underscores, etc.) and use the mapped NVR to identify the build. The original, unmodified NSV would be recorded in the "extra" field of the build.
(Archives can only have one btype. builds btypes must match their component archives. btype info can technically be {})
Is that sufficient to solve this mapping problem? Are there any other outstanding questions you need resolved, stano?
(Sorry for this getting splintered across multiple lists...)
So: * I'll change "modulemd" btype for the build to just "module" * I can do the tweak for n-v-r dashes fairly easily (i.e. convert - to _ in the version/stream field) and provide an mbs stream/version metadata in the extra/typeinfo/module space * As mentioned above - the output archives need the type too. Right now the "type" is "module" and there's an empty typeinfo for it. Should we use some generic type (i.e. "archive" or whatever makes sense) and just provide the "extra/typeinfo" piece? This is a little confusing since OSBS doesn't seem to be currently using this structure so I had no example to base from and the docs[1] are not making it clear. Basically the above is based on experimentation based on errors I was getting from koji when I was importing. I'd appreciate someone from koji folks to have a look and tell me whether I can likely simplify/adjust something.
Sorry about the fpaste expiring. I didn't realize the TTL is so short there. I uploaded the latest version example updated with changes proposed here: https://fedorapeople.org/~sochotni/mbs-content-generator.json
If the above content generator example seems OK I can do the final tweaks pretty quickly. I'd appreciate an ACK on the format from koji devs before I go ahead with that tweak though.
On 05/09/2017 07:45 AM, Stanislav Ochotnicky wrote:
On Fri 05 May 2017 07:23:18 PM CEST Ralph Bean rbean@redhat.com wrote:
On Mon, Apr 24, 2017 at 05:06:47PM +0200, Stanislav Ochotnicky wrote:
On Mon 24 Apr 2017 04:56:05 PM CEST Jan Kaluža jkaluza@redhat.com wrote:
On 04/24/2017 04:20 PM, Stanislav Ochotnicky wrote:
On Mon 24 Apr 2017 03:25:48 PM CEST Ralph Bean rbean@redhat.com wrote:
On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote: > This is the first stab and what the mbs CG metadata output might look like: > https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRLivL...
Just to clarify - in the "build" entry here, this is a build of the "nginx module". The "version" (which is 1) and the "release" (which is 2) are in koji terms, not MBS terms.
The mapping between MBS and koji terms should look something like:
MBS -> Koji --------------- name -> name stream -> version version -> releaseThe only kicker I can see at the moment is the datatype of stream(MBS) versus version(Koji). The MBS stream is a string (master, or f26, or whatever). The Koji version... does it have to be an integer?
Well the version even for rpms is a string. Usually semantic versioning style thing - but not necessarily. Of course if you use something lime rpmdev-vercmp on these mbs builds EVRs it might make things weird. I.e.: rpmdev-vercmp master-20 version2-10 master-20 < version2-10
But I think that's OK.
One thing however - can stream contain dash? Because version string cannot. I'd be tempted to just replace it with underscore hoping there won't be both :-)
We currently set the the dist-git branch as a stream value. From MBS point of view, there is no problem with dash in a stream name and I presume there will dashes in a stream name if we don't decide you cannot use dash in dist-git branch name for modules. I would not like to restrict that if there is no real reason to do that.
How much user-visible will the content generator data be? Can for example user search for module in Koji by typing its name-stream-version? In that case, I would start thinking about restricting the branch so it does not contain dash, because people will not replace dash with underscore when searching Koji.
Well this part of content generator is very visible - it's part of the build NVR after all. So things like API calls getBuild(nvr) would be affected by it. And the NVR will need to be input as part of shipping update in Bodhi for example.
I would guess people *will* use dash in the stream name unless they are prevented from doing it?
From discussion in IRC today (paraphrasing others):
We can use a new "build type" for this ("module"). With a new build type (btype) we can:
- store the original MBS name-stream-version in koji as extra information, preserving dashes
- while the build has a koji-reformatted name-version-release with dashes converted to underscores.
There is precedent for this in the way Maven GAV is mapped to NVR.
In the CG metadata provided by MBS, btype specific data goes into extra->typeinfo->BTYPENAME for build and archive and this how koji knows the btype. So, the MBS would need to define the mapping from NSV to NVR (hyphens become underscores, etc.) and use the mapped NVR to identify the build. The original, unmodified NSV would be recorded in the "extra" field of the build.
(Archives can only have one btype. builds btypes must match their component archives. btype info can technically be {})
Is that sufficient to solve this mapping problem? Are there any other outstanding questions you need resolved, stano?
(Sorry for this getting splintered across multiple lists...)
So:
- I'll change "modulemd" btype for the build to just "module"
yep
- I can do the tweak for n-v-r dashes fairly easily (i.e. convert - to _ in the version/stream field) and provide an mbs stream/version metadata in the extra/typeinfo/module space
ok
- As mentioned above - the output archives need the type too. Right now the "type" is "module" and there's an empty typeinfo for it. Should we use some generic type (i.e. "archive" or whatever makes sense) and just provide the "extra/typeinfo" piece? This is a little confusing since OSBS doesn't seem to be currently using this structure so I had no example to base from and the docs[1] are not making it clear. Basically the above is based on experimentation based on errors I was getting from koji when I was importing. I'd appreciate someone from koji folks to have a look and tell me whether I can likely simplify/adjust something.
The btype given to the archive must match one of the btypes for the build. Unless I'm missing something, this should be "module" as well, unless you want to import a multi-typed build.
Unfortunately, the term type is way overloaded here, hence my use of 'btype'.
Regarding how osbs does it... btypes are fairly new. They were added to allow CGs to import new types of builds without having to change Koji. The old "legacy" types (rpm, maven, win, image) are also btypes now, but optionally those support the old way of specifying the typeinfo (directly in extra). Since the containerbuild plugin for osbs predates btypes and only imports image builds, it still uses the old way.
Sorry about the fpaste expiring. I didn't realize the TTL is so short there. I uploaded the latest version example updated with changes proposed here: https://fedorapeople.org/~sochotni/mbs-content-generator.json
buildroot entry is very sparse. no components or tools listed
buildroots.0.container.type = "none"
I take it that means no container at all, just a service host making things?
output.0.type should be "file" the btype info for the artifact should go in output.0.extra.typeinfo
...gotta run. I'll look more later
If the above content generator example seems OK I can do the final tweaks pretty quickly. I'd appreciate an ACK on the format from koji devs before I go ahead with that tweak though.
Just a quick repeat from the other email in the thread. PR with these changes: https://pagure.io/fm-orchestrator/pull-request/554 Example output: https://fedorapeople.org/~sochotni/mbs-content-generator-v2.json
Summary of changes from original: * Added modulemd version to tools * Added all installed rpms from mbs host into the buildroots.0.components * Changed output.0.type from "modulemd" to "file" * Added empty extra.typeinfo.module into output.0
On Fri 12 May 2017 04:55:21 PM CEST Mike McLean wrote:
So:
- I'll change "modulemd" btype for the build to just "module"
yep
Done
- I can do the tweak for n-v-r dashes fairly easily (i.e. convert - to _ in the version/stream field) and provide an mbs stream/version metadata in the extra/typeinfo/module space
ok
Done
- As mentioned above - the output archives need the type too. Right now the "type" is "module" and there's an empty typeinfo for it. Should we use some generic type (i.e. "archive" or whatever makes sense) and just provide the "extra/typeinfo" piece? This is a little confusing since OSBS doesn't seem to be currently using this structure so I had no example to base from and the docs[1] are not making it clear. Basically the above is based on experimentation based on errors I was getting from koji when I was importing. I'd appreciate someone from koji folks to have a look and tell me whether I can likely simplify/adjust something.
The btype given to the archive must match one of the btypes for the build. Unless I'm missing something, this should be "module" as well, unless you want to import a multi-typed build.
I believe done
Unfortunately, the term type is way overloaded here, hence my use of 'btype'.
Regarding how osbs does it... btypes are fairly new. They were added to allow CGs to import new types of builds without having to change Koji. The old "legacy" types (rpm, maven, win, image) are also btypes now, but optionally those support the old way of specifying the typeinfo (directly in extra). Since the containerbuild plugin for osbs predates btypes and only imports image builds, it still uses the old way.
Ah, yeah that makes it clearer. Thanks for clarification.
Sorry about the fpaste expiring. I didn't realize the TTL is so short there. I uploaded the latest version example updated with changes proposed here: https://fedorapeople.org/~sochotni/mbs-content-generator.json
buildroot entry is very sparse. no components or tools listed
Fixed - added modulemd version into the tools. And components now list all installed rpms on the mbs host system. MBS just "orchestrates" koji so technically there aren't many things that should affect the behaviour. If we think of more - we can add them to the tools.
buildroots.0.container.type = "none"
I take it that means no container at all, just a service host making things?
Correct. No container used (I'll note that requiring the architecture is a bit weird in this case but not a big deal).
output.0.type should be "file" the btype info for the artifact should go in output.0.extra.typeinfo
I think this is now fixed
...gotta run. I'll look more later
Thanks
Just pinging on this. I think stano is blocked waiting for an ack from a koji-devel and (possibly) fedora-releng here.
On Tue, May 09, 2017 at 01:45:55PM +0200, Stanislav Ochotnicky wrote:
On Fri 05 May 2017 07:23:18 PM CEST Ralph Bean rbean@redhat.com wrote:
On Mon, Apr 24, 2017 at 05:06:47PM +0200, Stanislav Ochotnicky wrote:
On Mon 24 Apr 2017 04:56:05 PM CEST Jan Kaluža jkaluza@redhat.com wrote:
On 04/24/2017 04:20 PM, Stanislav Ochotnicky wrote:
On Mon 24 Apr 2017 03:25:48 PM CEST Ralph Bean rbean@redhat.com wrote:
On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote: > This is the first stab and what the mbs CG metadata output might look like: > https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRLivL...
Just to clarify - in the "build" entry here, this is a build of the "nginx module". The "version" (which is 1) and the "release" (which is 2) are in koji terms, not MBS terms.
The mapping between MBS and koji terms should look something like:
MBS -> Koji --------------- name -> name stream -> version version -> releaseThe only kicker I can see at the moment is the datatype of stream(MBS) versus version(Koji). The MBS stream is a string (master, or f26, or whatever). The Koji version... does it have to be an integer?
Well the version even for rpms is a string. Usually semantic versioning style thing - but not necessarily. Of course if you use something lime rpmdev-vercmp on these mbs builds EVRs it might make things weird. I.e.: rpmdev-vercmp master-20 version2-10 master-20 < version2-10
But I think that's OK.
One thing however - can stream contain dash? Because version string cannot. I'd be tempted to just replace it with underscore hoping there won't be both :-)
We currently set the the dist-git branch as a stream value. From MBS point of view, there is no problem with dash in a stream name and I presume there will dashes in a stream name if we don't decide you cannot use dash in dist-git branch name for modules. I would not like to restrict that if there is no real reason to do that.
How much user-visible will the content generator data be? Can for example user search for module in Koji by typing its name-stream-version? In that case, I would start thinking about restricting the branch so it does not contain dash, because people will not replace dash with underscore when searching Koji.
Well this part of content generator is very visible - it's part of the build NVR after all. So things like API calls getBuild(nvr) would be affected by it. And the NVR will need to be input as part of shipping update in Bodhi for example.
I would guess people *will* use dash in the stream name unless they are prevented from doing it?
From discussion in IRC today (paraphrasing others):
We can use a new "build type" for this ("module"). With a new build type (btype) we can:
- store the original MBS name-stream-version in koji as extra information, preserving dashes
- while the build has a koji-reformatted name-version-release with dashes converted to underscores.
There is precedent for this in the way Maven GAV is mapped to NVR.
In the CG metadata provided by MBS, btype specific data goes into extra->typeinfo->BTYPENAME for build and archive and this how koji knows the btype. So, the MBS would need to define the mapping from NSV to NVR (hyphens become underscores, etc.) and use the mapped NVR to identify the build. The original, unmodified NSV would be recorded in the "extra" field of the build.
(Archives can only have one btype. builds btypes must match their component archives. btype info can technically be {})
Is that sufficient to solve this mapping problem? Are there any other outstanding questions you need resolved, stano?
(Sorry for this getting splintered across multiple lists...)
So:
- I'll change "modulemd" btype for the build to just "module"
- I can do the tweak for n-v-r dashes fairly easily (i.e. convert - to _ in the version/stream field) and provide an mbs stream/version metadata in the extra/typeinfo/module space
- As mentioned above - the output archives need the type too. Right now the "type" is "module" and there's an empty typeinfo for it. Should we use some generic type (i.e. "archive" or whatever makes sense) and just provide the "extra/typeinfo" piece? This is a little confusing since OSBS doesn't seem to be currently using this structure so I had no example to base from and the docs[1] are not making it clear. Basically the above is based on experimentation based on errors I was getting from koji when I was importing. I'd appreciate someone from koji folks to have a look and tell me whether I can likely simplify/adjust something.
Sorry about the fpaste expiring. I didn't realize the TTL is so short there. I uploaded the latest version example updated with changes proposed here: https://fedorapeople.org/~sochotni/mbs-content-generator.json
If the above content generator example seems OK I can do the final tweaks pretty quickly. I'd appreciate an ACK on the format from koji devs before I go ahead with that tweak though.
On 05/12/2017 01:07 PM, Ralph Bean wrote:
Just pinging on this. I think stano is blocked waiting for an ack from a koji-devel and (possibly) fedora-releng here.
Overall thrust is fine. Particulars of the data need work. We need to make sure to distinguish between:
btype: the build type specified for builds and archives (e.g. rpm, maven, win, image, module). That is the data that goes into typeinfo.
output type: this unfortunately named field in the metadata should be either "file", "rpm", or "log".
archivetype: not specified in the metadata, but reported in the api based on file extension (e.g. jar, zip, pom, iso, qcow2)
On Tue, May 09, 2017 at 01:45:55PM +0200, Stanislav Ochotnicky wrote:
On Fri 05 May 2017 07:23:18 PM CEST Ralph Bean rbean@redhat.com wrote:
On Mon, Apr 24, 2017 at 05:06:47PM +0200, Stanislav Ochotnicky wrote:
On Mon 24 Apr 2017 04:56:05 PM CEST Jan Kaluža jkaluza@redhat.com wrote:
On 04/24/2017 04:20 PM, Stanislav Ochotnicky wrote:
On Mon 24 Apr 2017 03:25:48 PM CEST Ralph Bean rbean@redhat.com wrote: > On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote: >> This is the first stab and what the mbs CG metadata output might look like: >> https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRLivL... > > Just to clarify - in the "build" entry here, this is a build of the > "nginx module". The "version" (which is 1) and the "release" (which is > 2) are in koji terms, not MBS terms. > > The mapping between MBS and koji terms should look something like: > > MBS -> Koji > --------------- > name -> name > stream -> version > version -> release > > The only kicker I can see at the moment is the datatype of stream(MBS) > versus version(Koji). The MBS stream is a string (master, or f26, or > whatever). The Koji version... does it have to be an integer?
Well the version even for rpms is a string. Usually semantic versioning style thing - but not necessarily. Of course if you use something lime rpmdev-vercmp on these mbs builds EVRs it might make things weird. I.e.: rpmdev-vercmp master-20 version2-10 master-20 < version2-10
But I think that's OK.
One thing however - can stream contain dash? Because version string cannot. I'd be tempted to just replace it with underscore hoping there won't be both :-)
We currently set the the dist-git branch as a stream value. From MBS point of view, there is no problem with dash in a stream name and I presume there will dashes in a stream name if we don't decide you cannot use dash in dist-git branch name for modules. I would not like to restrict that if there is no real reason to do that.
How much user-visible will the content generator data be? Can for example user search for module in Koji by typing its name-stream-version? In that case, I would start thinking about restricting the branch so it does not contain dash, because people will not replace dash with underscore when searching Koji.
Well this part of content generator is very visible - it's part of the build NVR after all. So things like API calls getBuild(nvr) would be affected by it. And the NVR will need to be input as part of shipping update in Bodhi for example.
I would guess people *will* use dash in the stream name unless they are prevented from doing it?
From discussion in IRC today (paraphrasing others):
We can use a new "build type" for this ("module"). With a new build type (btype) we can:
- store the original MBS name-stream-version in koji as extra information, preserving dashes
- while the build has a koji-reformatted name-version-release with dashes converted to underscores.
There is precedent for this in the way Maven GAV is mapped to NVR.
In the CG metadata provided by MBS, btype specific data goes into extra->typeinfo->BTYPENAME for build and archive and this how koji knows the btype. So, the MBS would need to define the mapping from NSV to NVR (hyphens become underscores, etc.) and use the mapped NVR to identify the build. The original, unmodified NSV would be recorded in the "extra" field of the build.
(Archives can only have one btype. builds btypes must match their component archives. btype info can technically be {})
Is that sufficient to solve this mapping problem? Are there any other outstanding questions you need resolved, stano?
(Sorry for this getting splintered across multiple lists...)
So:
- I'll change "modulemd" btype for the build to just "module"
- I can do the tweak for n-v-r dashes fairly easily (i.e. convert - to _ in the version/stream field) and provide an mbs stream/version metadata in the extra/typeinfo/module space
- As mentioned above - the output archives need the type too. Right now the "type" is "module" and there's an empty typeinfo for it. Should we use some generic type (i.e. "archive" or whatever makes sense) and just provide the "extra/typeinfo" piece? This is a little confusing since OSBS doesn't seem to be currently using this structure so I had no example to base from and the docs[1] are not making it clear. Basically the above is based on experimentation based on errors I was getting from koji when I was importing. I'd appreciate someone from koji folks to have a look and tell me whether I can likely simplify/adjust something.
Sorry about the fpaste expiring. I didn't realize the TTL is so short there. I uploaded the latest version example updated with changes proposed here: https://fedorapeople.org/~sochotni/mbs-content-generator.json
If the above content generator example seems OK I can do the final tweaks pretty quickly. I'd appreciate an ACK on the format from koji devs before I go ahead with that tweak though.
koji-devel mailing list -- koji-devel@lists.fedorahosted.org To unsubscribe send an email to koji-devel-leave@lists.fedorahosted.org
I did make tweaks to the code representing these changes in: https://pagure.io/fm-orchestrator/pull-request/554
Example output: https://fedorapeople.org/~sochotni/mbs-content-generator-v2.json
Summary of changes from original: * Added modulemd version to tools * Added all installed rpms from mbs host into the buildroots.0.components * Changed output.0.type from "modulemd" to "file" * Added empty extra.typeinfo.module into output.0
Few more comments inline
On Fri 12 May 2017 07:43:46 PM CEST Mike McLean wrote:
On 05/12/2017 01:07 PM, Ralph Bean wrote:
Just pinging on this. I think stano is blocked waiting for an ack from a koji-devel and (possibly) fedora-releng here.
Overall thrust is fine. Particulars of the data need work. We need to make sure to distinguish between:
btype: the build type specified for builds and archives (e.g. rpm, maven, win, image, module). That is the data that goes into typeinfo.
OK, this should be better now
output type: this unfortunately named field in the metadata should be either "file", "rpm", or "log".
Changed the output type to file
archivetype: not specified in the metadata, but reported in the api based on file extension (e.g. jar, zip, pom, iso, qcow2)
So the file is a yaml metadata file. Not an archive really. Which is what my initial question around this was - we use this metadata file as a place to link those components described by the metadata file. Is that fine? I am guessing we'll need to add "yaml" extension to archivetype in the db?
On Tue, May 09, 2017 at 01:45:55PM +0200, Stanislav Ochotnicky wrote:
On Fri 05 May 2017 07:23:18 PM CEST Ralph Bean rbean@redhat.com wrote:
On Mon, Apr 24, 2017 at 05:06:47PM +0200, Stanislav Ochotnicky wrote:
On Mon 24 Apr 2017 04:56:05 PM CEST Jan Kaluža jkaluza@redhat.com wrote:
On 04/24/2017 04:20 PM, Stanislav Ochotnicky wrote: > On Mon 24 Apr 2017 03:25:48 PM CEST Ralph Bean rbean@redhat.com wrote: >> On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote: >>> This is the first stab and what the mbs CG metadata output might look like: >>> https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRLivL... >> >> Just to clarify - in the "build" entry here, this is a build of the >> "nginx module". The "version" (which is 1) and the "release" (which is >> 2) are in koji terms, not MBS terms. >> >> The mapping between MBS and koji terms should look something like: >> >> MBS -> Koji >> --------------- >> name -> name >> stream -> version >> version -> release >> >> The only kicker I can see at the moment is the datatype of stream(MBS) >> versus version(Koji). The MBS stream is a string (master, or f26, or >> whatever). The Koji version... does it have to be an integer? > > Well the version even for rpms is a string. Usually semantic versioning > style thing - but not necessarily. Of course if you use something lime > rpmdev-vercmp on these mbs builds EVRs it might make things weird. I.e.: > rpmdev-vercmp master-20 version2-10 > master-20 < version2-10 > > But I think that's OK. > > One thing however - can stream contain dash? Because version string > cannot. I'd be tempted to just replace it with underscore hoping there > won't be both :-)
We currently set the the dist-git branch as a stream value. From MBS point of view, there is no problem with dash in a stream name and I presume there will dashes in a stream name if we don't decide you cannot use dash in dist-git branch name for modules. I would not like to restrict that if there is no real reason to do that.
How much user-visible will the content generator data be? Can for example user search for module in Koji by typing its name-stream-version? In that case, I would start thinking about restricting the branch so it does not contain dash, because people will not replace dash with underscore when searching Koji.
Well this part of content generator is very visible - it's part of the build NVR after all. So things like API calls getBuild(nvr) would be affected by it. And the NVR will need to be input as part of shipping update in Bodhi for example.
I would guess people *will* use dash in the stream name unless they are prevented from doing it?
From discussion in IRC today (paraphrasing others):
We can use a new "build type" for this ("module"). With a new build type (btype) we can:
- store the original MBS name-stream-version in koji as extra information, preserving dashes
- while the build has a koji-reformatted name-version-release with dashes converted to underscores.
There is precedent for this in the way Maven GAV is mapped to NVR.
In the CG metadata provided by MBS, btype specific data goes into extra->typeinfo->BTYPENAME for build and archive and this how koji knows the btype. So, the MBS would need to define the mapping from NSV to NVR (hyphens become underscores, etc.) and use the mapped NVR to identify the build. The original, unmodified NSV would be recorded in the "extra" field of the build.
(Archives can only have one btype. builds btypes must match their component archives. btype info can technically be {})
Is that sufficient to solve this mapping problem? Are there any other outstanding questions you need resolved, stano?
(Sorry for this getting splintered across multiple lists...)
So:
- I'll change "modulemd" btype for the build to just "module"
- I can do the tweak for n-v-r dashes fairly easily (i.e. convert - to _ in the version/stream field) and provide an mbs stream/version metadata in the extra/typeinfo/module space
- As mentioned above - the output archives need the type too. Right now the "type" is "module" and there's an empty typeinfo for it. Should we use some generic type (i.e. "archive" or whatever makes sense) and just provide the "extra/typeinfo" piece? This is a little confusing since OSBS doesn't seem to be currently using this structure so I had no example to base from and the docs[1] are not making it clear. Basically the above is based on experimentation based on errors I was getting from koji when I was importing. I'd appreciate someone from koji folks to have a look and tell me whether I can likely simplify/adjust something.
Sorry about the fpaste expiring. I didn't realize the TTL is so short there. I uploaded the latest version example updated with changes proposed here: https://fedorapeople.org/~sochotni/mbs-content-generator.json
If the above content generator example seems OK I can do the final tweaks pretty quickly. I'd appreciate an ACK on the format from koji devs before I go ahead with that tweak though.
koji-devel mailing list -- koji-devel@lists.fedorahosted.org To unsubscribe send an email to koji-devel-leave@lists.fedorahosted.org
koji-devel mailing list -- koji-devel@lists.fedorahosted.org To unsubscribe send an email to koji-devel-leave@lists.fedorahosted.org
I have added a request for koji in Fedora infrastructure to add configuration to enable all of this. Could you folks check if I forgot anything and/or clarify what I requested?
https://pagure.io/releng/issue/6799
Thanks!
On Mon 15 May 2017 12:21:17 PM CEST Stanislav Ochotnicky sochotnicky@redhat.com wrote:
I did make tweaks to the code representing these changes in: https://pagure.io/fm-orchestrator/pull-request/554
Example output: https://fedorapeople.org/~sochotni/mbs-content-generator-v2.json
Summary of changes from original:
- Added modulemd version to tools
- Added all installed rpms from mbs host into the buildroots.0.components
- Changed output.0.type from "modulemd" to "file"
- Added empty extra.typeinfo.module into output.0
Few more comments inline
On Fri 12 May 2017 07:43:46 PM CEST Mike McLean wrote:
On 05/12/2017 01:07 PM, Ralph Bean wrote:
Just pinging on this. I think stano is blocked waiting for an ack from a koji-devel and (possibly) fedora-releng here.
Overall thrust is fine. Particulars of the data need work. We need to make sure to distinguish between:
btype: the build type specified for builds and archives (e.g. rpm, maven, win, image, module). That is the data that goes into typeinfo.
OK, this should be better now
output type: this unfortunately named field in the metadata should be either "file", "rpm", or "log".
Changed the output type to file
archivetype: not specified in the metadata, but reported in the api based on file extension (e.g. jar, zip, pom, iso, qcow2)
So the file is a yaml metadata file. Not an archive really. Which is what my initial question around this was - we use this metadata file as a place to link those components described by the metadata file. Is that fine? I am guessing we'll need to add "yaml" extension to archivetype in the db?
On Tue, May 09, 2017 at 01:45:55PM +0200, Stanislav Ochotnicky wrote:
On Fri 05 May 2017 07:23:18 PM CEST Ralph Bean rbean@redhat.com wrote:
On Mon, Apr 24, 2017 at 05:06:47PM +0200, Stanislav Ochotnicky wrote:
On Mon 24 Apr 2017 04:56:05 PM CEST Jan Kaluža jkaluza@redhat.com wrote: > On 04/24/2017 04:20 PM, Stanislav Ochotnicky wrote: >> On Mon 24 Apr 2017 03:25:48 PM CEST Ralph Bean rbean@redhat.com wrote: >>> On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote: >>>> This is the first stab and what the mbs CG metadata output might look like: >>>> https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRLivL... >>> >>> Just to clarify - in the "build" entry here, this is a build of the >>> "nginx module". The "version" (which is 1) and the "release" (which is >>> 2) are in koji terms, not MBS terms. >>> >>> The mapping between MBS and koji terms should look something like: >>> >>> MBS -> Koji >>> --------------- >>> name -> name >>> stream -> version >>> version -> release >>> >>> The only kicker I can see at the moment is the datatype of stream(MBS) >>> versus version(Koji). The MBS stream is a string (master, or f26, or >>> whatever). The Koji version... does it have to be an integer? >> >> Well the version even for rpms is a string. Usually semantic versioning >> style thing - but not necessarily. Of course if you use something lime >> rpmdev-vercmp on these mbs builds EVRs it might make things weird. I.e.: >> rpmdev-vercmp master-20 version2-10 >> master-20 < version2-10 >> >> But I think that's OK. >> >> One thing however - can stream contain dash? Because version string >> cannot. I'd be tempted to just replace it with underscore hoping there >> won't be both :-) > > We currently set the the dist-git branch as a stream value. From MBS > point of view, there is no problem with dash in a stream name and I > presume there will dashes in a stream name if we don't decide you cannot > use dash in dist-git branch name for modules. I would not like to > restrict that if there is no real reason to do that. > > How much user-visible will the content generator data be? Can for > example user search for module in Koji by typing its > name-stream-version? In that case, I would start thinking about > restricting the branch so it does not contain dash, because people will > not replace dash with underscore when searching Koji.
Well this part of content generator is very visible - it's part of the build NVR after all. So things like API calls getBuild(nvr) would be affected by it. And the NVR will need to be input as part of shipping update in Bodhi for example.
I would guess people *will* use dash in the stream name unless they are prevented from doing it?
From discussion in IRC today (paraphrasing others):
We can use a new "build type" for this ("module"). With a new build type (btype) we can:
- store the original MBS name-stream-version in koji as extra information, preserving dashes
- while the build has a koji-reformatted name-version-release with dashes converted to underscores.
There is precedent for this in the way Maven GAV is mapped to NVR.
In the CG metadata provided by MBS, btype specific data goes into extra->typeinfo->BTYPENAME for build and archive and this how koji knows the btype. So, the MBS would need to define the mapping from NSV to NVR (hyphens become underscores, etc.) and use the mapped NVR to identify the build. The original, unmodified NSV would be recorded in the "extra" field of the build.
(Archives can only have one btype. builds btypes must match their component archives. btype info can technically be {})
Is that sufficient to solve this mapping problem? Are there any other outstanding questions you need resolved, stano?
(Sorry for this getting splintered across multiple lists...)
So:
- I'll change "modulemd" btype for the build to just "module"
- I can do the tweak for n-v-r dashes fairly easily (i.e. convert - to _ in the version/stream field) and provide an mbs stream/version metadata in the extra/typeinfo/module space
- As mentioned above - the output archives need the type too. Right now the "type" is "module" and there's an empty typeinfo for it. Should we use some generic type (i.e. "archive" or whatever makes sense) and just provide the "extra/typeinfo" piece? This is a little confusing since OSBS doesn't seem to be currently using this structure so I had no example to base from and the docs[1] are not making it clear. Basically the above is based on experimentation based on errors I was getting from koji when I was importing. I'd appreciate someone from koji folks to have a look and tell me whether I can likely simplify/adjust something.
Sorry about the fpaste expiring. I didn't realize the TTL is so short there. I uploaded the latest version example updated with changes proposed here: https://fedorapeople.org/~sochotni/mbs-content-generator.json
If the above content generator example seems OK I can do the final tweaks pretty quickly. I'd appreciate an ACK on the format from koji devs before I go ahead with that tweak though.
koji-devel mailing list -- koji-devel@lists.fedorahosted.org To unsubscribe send an email to koji-devel-leave@lists.fedorahosted.org
koji-devel mailing list -- koji-devel@lists.fedorahosted.org To unsubscribe send an email to koji-devel-leave@lists.fedorahosted.org
-- Stanislav Ochotnicky sochotnicky@redhat.com Business System Analyst, PnT DevOps - Brno
PGP: F434 2286 27DC 7D9B 2B64 0866 BCBD 752E 7B08 7241 Red Hat Inc. http://www.redhat.com
koji-devel@lists.fedorahosted.org
-
Mike McLean -
Ralph Bean -
Stanislav Ochotnicky