RFC: Content generator for module build service (v2)
by Stanislav Ochotnicky
Hi folks,
Note: my first email was sent to @fedoraproject.org so sorry for
duplication for some of you. This has been tweaked so the output is
actually importable into koji test instance with minor tweaks.
I'm working on module build service change that would make it into a
content generator similar to what OSBS is doing.
Quick intro around current state:
* MBS orchestrates koji builds, tags, etc
* Koji AFAIK knows nothing about mbs itself
* after finishing the build - mbs creates a module-specific tag and
tags every relevant build into it
This is the second stab and what the mbs CG metadata output might look
like:
https://paste.fedoraproject.org/paste/GM00iYtq55HXCQyXjbEVZV5M1UNdIGYhyRL...
I was actually able to import the above into a test koji instance (after
some SQL additions of new types etc).
Notable things:
* outputs - missing logs. Do we need them? Each module is basically a
collection (repository) of rpms built in koji which each have their
own logs. We could probably add orchestration logs later?
* tools, components - Not sure there's any set of specific tools worth
mentioning. Maybe version of koji client library?
* container - MBS makes no use of container so the "type" is none, but
we still have to specify the architecture or koji complains on import.
* "typeinfo" keys in build and output fields - I am a little
confused. When I tried to copy what OSBS is doing in the extra
fields koji refused to import the metadata. Current metadata is
based on some experimentation and adjustments based on error
messages. I think some guidance on the structure of the types and
typeinfo(s) would be nice since they don't seem to be up to date
with documentation.
And most importantly: the modulemd type output
- this would be a modulemd.yaml file that contains the mmd content [1] once
downloaded
- component metadata list would include all the rpms that are tagged in
the final module tag - all architectures, subpackages etc
- notably no signatures would ever be mentioned here since they are
not fixed at the end of module build unlike container builds
There was an idea to split the modulemd into separate arch-specific
pieces, but it didn't bring anything useful to the table so was
scratched.
Would the above approach be generally acceptable? Is there any concern
about adding components for file that is just metadata? Anything else I
might have missed?
Thanks,
[1] can be seen as "modulemd" in
https://mbs.fedoraproject.org/module-build-service/1/module-builds/199?ve...
[2] WIP PR for the MBS change is here: https://pagure.io/fm-orchestrator/pull-request/522#
--
Stanislav Ochotnicky <sochotnicky(a)redhat.com>
Business System Analyst, PnT DevOps - Brno
PGP: F434 2286 27DC 7D9B 2B64 0866 BCBD 752E 7B08 7241
Red Hat Inc. http://www.redhat.com
7 years
Re: RFC: Content generator for module build service
by Ralph Bean
On Tue, Apr 25, 2017 at 04:08:00PM -0400, Dennis Gregorovic wrote:
>
>
> On 04/24/2017 09:25 AM, Ralph Bean wrote:
> > On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote:
> >> This is the first stab and what the mbs CG metadata output might look like:
> >> https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRL...
> >
> > Just to clarify - in the "build" entry here, this is a build of the
> > "nginx module". The "version" (which is 1) and the "release" (which is
> > 2) are in koji terms, not MBS terms.
> >
> > The mapping between MBS and koji terms should look something like:
> >
> > MBS -> Koji
> > ---------------
> > name -> name
> > stream -> version
> > version -> release
> >
> > The only kicker I can see at the moment is the datatype of stream(MBS)
> > versus version(Koji). The MBS stream is a string (master, or f26, or
> > whatever). The Koji version... does it have to be an integer?
> >
>
> Is there potential for nvr conflict? For example if you have a nginx-1-2 RPM and a nginx-1-2 module
It is possible, but highly unlikely.
You would have to have:
- A traditional rpm with a 'release' like 201704250000 (unlikely)
- A module with a streamname like '1' (unlikely).
You're more likely to hit the second one, where the module streamname
might be "2.2", suggesting that the module is following the "2.2"
update stream of some major package. To get an NVR conflict, you
would need to hit both scenarios, though.
7 years
Re: RFC: Content generator for module build service
by Stanislav Ochotnicky
Just a quick repeat from the other email in the thread.
PR with these changes: https://pagure.io/fm-orchestrator/pull-request/554
Example output:
https://fedorapeople.org/~sochotni/mbs-content-generator-v2.json
Summary of changes from original:
* Added modulemd version to tools
* Added all installed rpms from mbs host into the buildroots.0.components
* Changed output.0.type from "modulemd" to "file"
* Added empty extra.typeinfo.module into output.0
On Fri 12 May 2017 04:55:21 PM CEST Mike McLean wrote:
>> So:
>> * I'll change "modulemd" btype for the build to just "module"
>
> yep
Done
>> * I can do the tweak for n-v-r dashes fairly easily (i.e. convert - to _
>> in the version/stream field) and provide an mbs stream/version metadata
>> in the extra/typeinfo/module space
>
> ok
Done
>> * As mentioned above - the output archives need the type too. Right now
>> the "type" is "module" and there's an empty typeinfo for it. Should
>> we use some generic type (i.e. "archive" or whatever makes sense) and
>> just provide the "extra/typeinfo" piece? This is a little confusing
>> since OSBS doesn't seem to be currently using this structure so I had
>> no example to base from and the docs[1] are not making it
>> clear. Basically the above is based on experimentation based on
>> errors I was getting from koji when I was importing. I'd appreciate
>> someone from koji folks to have a look and tell me whether I can
>> likely simplify/adjust something.
>
> The btype given to the archive must match one of the btypes for the
> build. Unless I'm missing something, this should be "module" as well,
> unless you want to import a multi-typed build.
I believe done
> Unfortunately, the term type is way overloaded here, hence my use of
> 'btype'.
>
> Regarding how osbs does it...
> btypes are fairly new. They were added to allow CGs to import new types
> of builds without having to change Koji. The old "legacy" types (rpm,
> maven, win, image) are also btypes now, but optionally those support the
> old way of specifying the typeinfo (directly in extra). Since the
> containerbuild plugin for osbs predates btypes and only imports image
> builds, it still uses the old way.
Ah, yeah that makes it clearer. Thanks for clarification.
>> Sorry about the fpaste expiring. I didn't realize the TTL is so short
>> there. I uploaded the latest version example updated with changes
>> proposed here:
>> https://fedorapeople.org/~sochotni/mbs-content-generator.json
>
> buildroot entry is very sparse. no components or tools listed
Fixed - added modulemd version into the tools. And components now list
all installed rpms on the mbs host system. MBS just "orchestrates"
koji so technically there aren't many things that should affect the
behaviour. If we think of more - we can add them to the tools.
> buildroots.0.container.type = "none"
>
> I take it that means no container at all, just a service host making things?
Correct. No container used (I'll note that requiring the architecture is
a bit weird in this case but not a big deal).
> output.0.type should be "file"
> the btype info for the artifact should go in output.0.extra.typeinfo
I think this is now fixed
> ...gotta run. I'll look more later
Thanks
--
Stanislav Ochotnicky <sochotnicky(a)redhat.com>
Business System Analyst, PnT DevOps - Brno
PGP: F434 2286 27DC 7D9B 2B64 0866 BCBD 752E 7B08 7241
Red Hat Inc. http://www.redhat.com
6 years, 11 months
Re: RFC: Content generator for module build service
by Dennis Gregorovic
On 04/24/2017 09:25 AM, Ralph Bean wrote:
> On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote:
>> This is the first stab and what the mbs CG metadata output might look like:
>> https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRL...
>
> Just to clarify - in the "build" entry here, this is a build of the
> "nginx module". The "version" (which is 1) and the "release" (which is
> 2) are in koji terms, not MBS terms.
>
> The mapping between MBS and koji terms should look something like:
>
> MBS -> Koji
> ---------------
> name -> name
> stream -> version
> version -> release
>
> The only kicker I can see at the moment is the datatype of stream(MBS)
> versus version(Koji). The MBS stream is a string (master, or f26, or
> whatever). The Koji version... does it have to be an integer?
>
Is there potential for nvr conflict? For example if you have a nginx-1-2 RPM and a nginx-1-2 module
7 years
Re: RFC: Content generator for module build service
by Mike Bonnet
Re-added koji-devel.
On 04/24/2017 06:25 AM, Ralph Bean wrote:
> On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote:
>> This is the first stab and what the mbs CG metadata output might look like:
>> https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRL...
>
> Just to clarify - in the "build" entry here, this is a build of the
> "nginx module". The "version" (which is 1) and the "release" (which is
> 2) are in koji terms, not MBS terms.
>
> The mapping between MBS and koji terms should look something like:
>
> MBS -> Koji
> ---------------
> name -> name
> stream -> version
> version -> release
>
> The only kicker I can see at the moment is the datatype of stream(MBS)
> versus version(Koji). The MBS stream is a string (master, or f26, or
> whatever). The Koji version... does it have to be an integer?
No, the Koji version and release are also strings. However, because of
the way NVRs are parsed, neither version nor release can contain a
hyphen (-). Not sure if MBS has the same constraint, but if not, we may
need some mapping between the two. Maybe a new build type (and
associated metadata).
7 years
Re: RFC: Content generator for module build service
by Ralph Bean
On Mon, Apr 24, 2017 at 05:06:47PM +0200, Stanislav Ochotnicky wrote:
> On Mon 24 Apr 2017 04:56:05 PM CEST Jan Kaluža <jkaluza(a)redhat.com> wrote:
> > On 04/24/2017 04:20 PM, Stanislav Ochotnicky wrote:
> >> On Mon 24 Apr 2017 03:25:48 PM CEST Ralph Bean <rbean(a)redhat.com> wrote:
> >>> On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote:
> >>>> This is the first stab and what the mbs CG metadata output might look like:
> >>>> https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRL...
> >>>
> >>> Just to clarify - in the "build" entry here, this is a build of the
> >>> "nginx module". The "version" (which is 1) and the "release" (which is
> >>> 2) are in koji terms, not MBS terms.
> >>>
> >>> The mapping between MBS and koji terms should look something like:
> >>>
> >>> MBS -> Koji
> >>> ---------------
> >>> name -> name
> >>> stream -> version
> >>> version -> release
> >>>
> >>> The only kicker I can see at the moment is the datatype of stream(MBS)
> >>> versus version(Koji). The MBS stream is a string (master, or f26, or
> >>> whatever). The Koji version... does it have to be an integer?
> >>
> >> Well the version even for rpms is a string. Usually semantic versioning
> >> style thing - but not necessarily. Of course if you use something lime
> >> rpmdev-vercmp on these mbs builds EVRs it might make things weird. I.e.:
> >> rpmdev-vercmp master-20 version2-10
> >> master-20 < version2-10
> >>
> >> But I think that's OK.
> >>
> >> One thing however - can stream contain dash? Because version string
> >> cannot. I'd be tempted to just replace it with underscore hoping there
> >> won't be both :-)
> >
> > We currently set the the dist-git branch as a stream value. From MBS
> > point of view, there is no problem with dash in a stream name and I
> > presume there will dashes in a stream name if we don't decide you cannot
> > use dash in dist-git branch name for modules. I would not like to
> > restrict that if there is no real reason to do that.
> >
> > How much user-visible will the content generator data be? Can for
> > example user search for module in Koji by typing its
> > name-stream-version? In that case, I would start thinking about
> > restricting the branch so it does not contain dash, because people will
> > not replace dash with underscore when searching Koji.
>
> Well this part of content generator is very visible - it's part of the
> build NVR after all. So things like API calls getBuild(nvr) would be
> affected by it. And the NVR will need to be input as part of shipping
> update in Bodhi for example.
>
> I would guess people *will* use dash in the stream name unless they are
> prevented from doing it?
From discussion in IRC today (paraphrasing others):
We can use a new "build type" for this ("module"). With a new build
type (btype) we can:
- store the original MBS name-stream-version in koji as extra information, preserving dashes
- while the build has a koji-reformatted name-version-release with dashes converted to underscores.
There is precedent for this in the way Maven GAV is mapped to NVR.
In the CG metadata provided by MBS, btype specific data goes into
extra->typeinfo->BTYPENAME for build and archive and this how koji
knows the btype. So, the MBS would need to define the mapping from NSV
to NVR (hyphens become underscores, etc.) and use the mapped NVR to
identify the build. The original, unmodified NSV would be recorded in
the "extra" field of the build.
(Archives can only have one btype. builds btypes must match their
component archives. btype info can technically be {})
Is that sufficient to solve this mapping problem? Are there any other
outstanding questions you need resolved, stano?
(Sorry for this getting splintered across multiple lists...)
6 years, 11 months
Re: RFC: Content generator for module build service
by Stanislav Ochotnicky
On Fri 05 May 2017 07:23:18 PM CEST Ralph Bean <rbean(a)redhat.com> wrote:
> On Mon, Apr 24, 2017 at 05:06:47PM +0200, Stanislav Ochotnicky wrote:
>> On Mon 24 Apr 2017 04:56:05 PM CEST Jan Kaluža <jkaluza(a)redhat.com> wrote:
>> > On 04/24/2017 04:20 PM, Stanislav Ochotnicky wrote:
>> >> On Mon 24 Apr 2017 03:25:48 PM CEST Ralph Bean <rbean(a)redhat.com> wrote:
>> >>> On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote:
>> >>>> This is the first stab and what the mbs CG metadata output might look like:
>> >>>> https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRL...
>> >>>
>> >>> Just to clarify - in the "build" entry here, this is a build of the
>> >>> "nginx module". The "version" (which is 1) and the "release" (which is
>> >>> 2) are in koji terms, not MBS terms.
>> >>>
>> >>> The mapping between MBS and koji terms should look something like:
>> >>>
>> >>> MBS -> Koji
>> >>> ---------------
>> >>> name -> name
>> >>> stream -> version
>> >>> version -> release
>> >>>
>> >>> The only kicker I can see at the moment is the datatype of stream(MBS)
>> >>> versus version(Koji). The MBS stream is a string (master, or f26, or
>> >>> whatever). The Koji version... does it have to be an integer?
>> >>
>> >> Well the version even for rpms is a string. Usually semantic versioning
>> >> style thing - but not necessarily. Of course if you use something lime
>> >> rpmdev-vercmp on these mbs builds EVRs it might make things weird. I.e.:
>> >> rpmdev-vercmp master-20 version2-10
>> >> master-20 < version2-10
>> >>
>> >> But I think that's OK.
>> >>
>> >> One thing however - can stream contain dash? Because version string
>> >> cannot. I'd be tempted to just replace it with underscore hoping there
>> >> won't be both :-)
>> >
>> > We currently set the the dist-git branch as a stream value. From MBS
>> > point of view, there is no problem with dash in a stream name and I
>> > presume there will dashes in a stream name if we don't decide you cannot
>> > use dash in dist-git branch name for modules. I would not like to
>> > restrict that if there is no real reason to do that.
>> >
>> > How much user-visible will the content generator data be? Can for
>> > example user search for module in Koji by typing its
>> > name-stream-version? In that case, I would start thinking about
>> > restricting the branch so it does not contain dash, because people will
>> > not replace dash with underscore when searching Koji.
>>
>> Well this part of content generator is very visible - it's part of the
>> build NVR after all. So things like API calls getBuild(nvr) would be
>> affected by it. And the NVR will need to be input as part of shipping
>> update in Bodhi for example.
>>
>> I would guess people *will* use dash in the stream name unless they are
>> prevented from doing it?
>
> From discussion in IRC today (paraphrasing others):
>
> We can use a new "build type" for this ("module"). With a new build
> type (btype) we can:
>
> - store the original MBS name-stream-version in koji as extra information, preserving dashes
> - while the build has a koji-reformatted name-version-release with dashes converted to underscores.
>
> There is precedent for this in the way Maven GAV is mapped to NVR.
>
> In the CG metadata provided by MBS, btype specific data goes into
> extra->typeinfo->BTYPENAME for build and archive and this how koji
> knows the btype. So, the MBS would need to define the mapping from NSV
> to NVR (hyphens become underscores, etc.) and use the mapped NVR to
> identify the build. The original, unmodified NSV would be recorded in
> the "extra" field of the build.
>
> (Archives can only have one btype. builds btypes must match their
> component archives. btype info can technically be {})
>
> Is that sufficient to solve this mapping problem? Are there any other
> outstanding questions you need resolved, stano?
>
> (Sorry for this getting splintered across multiple lists...)
So:
* I'll change "modulemd" btype for the build to just "module"
* I can do the tweak for n-v-r dashes fairly easily (i.e. convert - to _
in the version/stream field) and provide an mbs stream/version metadata
in the extra/typeinfo/module space
* As mentioned above - the output archives need the type too. Right now
the "type" is "module" and there's an empty typeinfo for it. Should
we use some generic type (i.e. "archive" or whatever makes sense) and
just provide the "extra/typeinfo" piece? This is a little confusing
since OSBS doesn't seem to be currently using this structure so I had
no example to base from and the docs[1] are not making it
clear. Basically the above is based on experimentation based on
errors I was getting from koji when I was importing. I'd appreciate
someone from koji folks to have a look and tell me whether I can
likely simplify/adjust something.
Sorry about the fpaste expiring. I didn't realize the TTL is so short
there. I uploaded the latest version example updated with changes
proposed here:
https://fedorapeople.org/~sochotni/mbs-content-generator.json
If the above content generator example seems OK I can do the final
tweaks pretty quickly. I'd appreciate an ACK on the format from koji
devs before I go ahead with that tweak though.
--
Stanislav Ochotnicky <sochotnicky(a)redhat.com>
Business System Analyst, PnT DevOps - Brno
PGP: F434 2286 27DC 7D9B 2B64 0866 BCBD 752E 7B08 7241
Red Hat Inc. http://www.redhat.com
6 years, 11 months
Re: RFC: Content generator for module build service
by Mike McLean
On 05/09/2017 07:45 AM, Stanislav Ochotnicky wrote:
> On Fri 05 May 2017 07:23:18 PM CEST Ralph Bean <rbean(a)redhat.com> wrote:
>> On Mon, Apr 24, 2017 at 05:06:47PM +0200, Stanislav Ochotnicky wrote:
>>> On Mon 24 Apr 2017 04:56:05 PM CEST Jan Kaluža <jkaluza(a)redhat.com> wrote:
>>>> On 04/24/2017 04:20 PM, Stanislav Ochotnicky wrote:
>>>>> On Mon 24 Apr 2017 03:25:48 PM CEST Ralph Bean <rbean(a)redhat.com> wrote:
>>>>>> On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote:
>>>>>>> This is the first stab and what the mbs CG metadata output might look like:
>>>>>>> https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRL...
>>>>>>
>>>>>> Just to clarify - in the "build" entry here, this is a build of the
>>>>>> "nginx module". The "version" (which is 1) and the "release" (which is
>>>>>> 2) are in koji terms, not MBS terms.
>>>>>>
>>>>>> The mapping between MBS and koji terms should look something like:
>>>>>>
>>>>>> MBS -> Koji
>>>>>> ---------------
>>>>>> name -> name
>>>>>> stream -> version
>>>>>> version -> release
>>>>>>
>>>>>> The only kicker I can see at the moment is the datatype of stream(MBS)
>>>>>> versus version(Koji). The MBS stream is a string (master, or f26, or
>>>>>> whatever). The Koji version... does it have to be an integer?
>>>>>
>>>>> Well the version even for rpms is a string. Usually semantic versioning
>>>>> style thing - but not necessarily. Of course if you use something lime
>>>>> rpmdev-vercmp on these mbs builds EVRs it might make things weird. I.e.:
>>>>> rpmdev-vercmp master-20 version2-10
>>>>> master-20 < version2-10
>>>>>
>>>>> But I think that's OK.
>>>>>
>>>>> One thing however - can stream contain dash? Because version string
>>>>> cannot. I'd be tempted to just replace it with underscore hoping there
>>>>> won't be both :-)
>>>>
>>>> We currently set the the dist-git branch as a stream value. From MBS
>>>> point of view, there is no problem with dash in a stream name and I
>>>> presume there will dashes in a stream name if we don't decide you cannot
>>>> use dash in dist-git branch name for modules. I would not like to
>>>> restrict that if there is no real reason to do that.
>>>>
>>>> How much user-visible will the content generator data be? Can for
>>>> example user search for module in Koji by typing its
>>>> name-stream-version? In that case, I would start thinking about
>>>> restricting the branch so it does not contain dash, because people will
>>>> not replace dash with underscore when searching Koji.
>>>
>>> Well this part of content generator is very visible - it's part of the
>>> build NVR after all. So things like API calls getBuild(nvr) would be
>>> affected by it. And the NVR will need to be input as part of shipping
>>> update in Bodhi for example.
>>>
>>> I would guess people *will* use dash in the stream name unless they are
>>> prevented from doing it?
>>
>> From discussion in IRC today (paraphrasing others):
>>
>> We can use a new "build type" for this ("module"). With a new build
>> type (btype) we can:
>>
>> - store the original MBS name-stream-version in koji as extra information, preserving dashes
>> - while the build has a koji-reformatted name-version-release with dashes converted to underscores.
>>
>> There is precedent for this in the way Maven GAV is mapped to NVR.
>>
>> In the CG metadata provided by MBS, btype specific data goes into
>> extra->typeinfo->BTYPENAME for build and archive and this how koji
>> knows the btype. So, the MBS would need to define the mapping from NSV
>> to NVR (hyphens become underscores, etc.) and use the mapped NVR to
>> identify the build. The original, unmodified NSV would be recorded in
>> the "extra" field of the build.
>>
>> (Archives can only have one btype. builds btypes must match their
>> component archives. btype info can technically be {})
>>
>> Is that sufficient to solve this mapping problem? Are there any other
>> outstanding questions you need resolved, stano?
>>
>> (Sorry for this getting splintered across multiple lists...)
>
> So:
> * I'll change "modulemd" btype for the build to just "module"
yep
> * I can do the tweak for n-v-r dashes fairly easily (i.e. convert - to _
> in the version/stream field) and provide an mbs stream/version metadata
> in the extra/typeinfo/module space
ok
> * As mentioned above - the output archives need the type too. Right now
> the "type" is "module" and there's an empty typeinfo for it. Should
> we use some generic type (i.e. "archive" or whatever makes sense) and
> just provide the "extra/typeinfo" piece? This is a little confusing
> since OSBS doesn't seem to be currently using this structure so I had
> no example to base from and the docs[1] are not making it
> clear. Basically the above is based on experimentation based on
> errors I was getting from koji when I was importing. I'd appreciate
> someone from koji folks to have a look and tell me whether I can
> likely simplify/adjust something.
The btype given to the archive must match one of the btypes for the
build. Unless I'm missing something, this should be "module" as well,
unless you want to import a multi-typed build.
Unfortunately, the term type is way overloaded here, hence my use of
'btype'.
Regarding how osbs does it...
btypes are fairly new. They were added to allow CGs to import new types
of builds without having to change Koji. The old "legacy" types (rpm,
maven, win, image) are also btypes now, but optionally those support the
old way of specifying the typeinfo (directly in extra). Since the
containerbuild plugin for osbs predates btypes and only imports image
builds, it still uses the old way.
> Sorry about the fpaste expiring. I didn't realize the TTL is so short
> there. I uploaded the latest version example updated with changes
> proposed here:
> https://fedorapeople.org/~sochotni/mbs-content-generator.json
buildroot entry is very sparse. no components or tools listed
buildroots.0.container.type = "none"
I take it that means no container at all, just a service host making things?
output.0.type should be "file"
the btype info for the artifact should go in output.0.extra.typeinfo
...gotta run. I'll look more later
> If the above content generator example seems OK I can do the final
> tweaks pretty quickly. I'd appreciate an ACK on the format from koji
> devs before I go ahead with that tweak though.
>
>
6 years, 11 months