Hi all,
If you maintain any application in Fedora Infra (or outside) that tries to parse things out of RPM names, you might be interested in this.
For those wondering where I've been spending most of my time the last few weeks: I've been deep in the internals of Bodhi fixing all kinds of issues I've found between it and Modularity (turns out enabling composing was just the beginning).
At some point, I was informed that for modules, they are throwing all kinds of old assumptions out of the door. One of those is the good old Name-Version-Release in RPM land.
Modules do not have an NVR, they have an NSVC: Name, Stream, Version, Context. Name is the name of the module ("nodejs"). Stream is a stream, like "6", "7" or "8", these are crucial for the parallel availability that modules are trying to accomplish. Version is a date/time stamp, used to indicate new versions in the same name-stream. Context is a hash of some information in the built module that makes it so that the nodejs-6-20170314 built for Fedora 28 has a different identifier than the one built for Fedora 29.
There are some additional fields for installed things (arch and profile), but those aren't really important for things trying to parse/show module names I think.
One of the interesting things they wanted to also allow: dashes in streams. As a consequence, when you get an N-S-V.C as modules are represented in Koji builds, doing a .rsplit('-', 2) will not give you Name, Stream, Version.Context per se. You could totally have a module called nodejs-my-stream-5-20170314.abcd, with name=nodejs, stream=my-stream-5, version=20170314, context=abcd. There is no way for you to independently figure out what the NSVC components are, you will need to ask Koji, and use its name, version and release fields (with name=name, version=stream, release=version(.context)).
Also as a consequence, modules should not be dash-separated in anything the users see. For user consumption, they are Name:Stream:Version:Context, so you may need to manually convert between one representation and the other if you need to look up in koji or other systems versus display to users.
Also, note that if you are touching older modules in Koji (e.g. because you go through all koji builds), modules before February 19th will be lacking the Context field. However, I've been told that those modules will not be used, so unless you or your app goes spelunking in koji (for fun and profit), you should never see those.
I hope that this is useful information for anyone else finding themselves having to parse NVRs/NSVs.
Regards, Patrick
Thanks Patrick,
Sounds like a herculean effort, you might want to send this to devel@ as a FYI, you can never be quite sure what community gets up to so a heads up there is likely useful for someone there too.
P
On Thu, 15 Mar 2018, 07:27 Patrick Uiterwijk, puiterwijk@redhat.com wrote:
Hi all,
If you maintain any application in Fedora Infra (or outside) that tries to parse things out of RPM names, you might be interested in this.
For those wondering where I've been spending most of my time the last few weeks: I've been deep in the internals of Bodhi fixing all kinds of issues I've found between it and Modularity (turns out enabling composing was just the beginning).
At some point, I was informed that for modules, they are throwing all kinds of old assumptions out of the door. One of those is the good old Name-Version-Release in RPM land.
Modules do not have an NVR, they have an NSVC: Name, Stream, Version, Context. Name is the name of the module ("nodejs"). Stream is a stream, like "6", "7" or "8", these are crucial for the parallel availability that modules are trying to accomplish. Version is a date/time stamp, used to indicate new versions in the same name-stream. Context is a hash of some information in the built module that makes it so that the nodejs-6-20170314 built for Fedora 28 has a different identifier than the one built for Fedora 29.
There are some additional fields for installed things (arch and profile), but those aren't really important for things trying to parse/show module names I think.
One of the interesting things they wanted to also allow: dashes in streams. As a consequence, when you get an N-S-V.C as modules are represented in Koji builds, doing a .rsplit('-', 2) will not give you Name, Stream, Version.Context per se. You could totally have a module called nodejs-my-stream-5-20170314.abcd, with name=nodejs, stream=my-stream-5, version=20170314, context=abcd. There is no way for you to independently figure out what the NSVC components are, you will need to ask Koji, and use its name, version and release fields (with name=name, version=stream, release=version(.context)).
Also as a consequence, modules should not be dash-separated in anything the users see. For user consumption, they are Name:Stream:Version:Context, so you may need to manually convert between one representation and the other if you need to look up in koji or other systems versus display to users.
Also, note that if you are touching older modules in Koji (e.g. because you go through all koji builds), modules before February 19th will be lacking the Context field. However, I've been told that those modules will not be used, so unless you or your app goes spelunking in koji (for fun and profit), you should never see those.
I hope that this is useful information for anyone else finding themselves having to parse NVRs/NSVs.
Regards, Patrick _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org
On Thu, Mar 15, 2018 at 03:26:50PM +0100, Patrick Uiterwijk wrote:
As a consequence, when you get an N-S-V.C as modules are represented in Koji builds, doing a .rsplit('-', 2) will not give you Name, Stream, Version.Context per se. You could totally have a module called nodejs-my-stream-5-20170314.abcd, with name=nodejs, stream=my-stream-5, version=20170314, context=abcd. There is no way for you to independently figure out what the NSVC components are, you will need to ask Koji, and use its name, version and release fields (with name=name, version=stream, release=version(.context)).
I really question this change and if we shouldn't fix it as this is going to break every single script trying to get package name from binary/build.
Thanks for all your work Patrick and the heads up about this.
Pierre
Please note that modules (and therefore module builds) may reference to multiple RPM builds. Also, the module name doesn't need to match any RPM name. Therefore getting package names from the module build name would not be possible anyway.
As an exapmple, see the following build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1057280
Build name: "golang-ecosystem-2017.0-20180312141905.40a0ab3e"
If you open the modulemd [1] (a module definition file) linked at the bottom of the page, you'll see that this module build contains four RPM components (SRPMs): * golang-github-russross-blackfriday * go-srpm-macros * golang-github-cpuguy83-go-md2man * go-compilers
The page also says "Package Name golang-ecosystem" which links to a "golang-ecosystem" package [2] in Koji. That's weird, there is no such package named "golang-ecosystem". The page lists two module builds, not two package builds.
Another example, where a module references only one RPM package and the module name matches the package name: https://koji.fedoraproject.org/koji/buildinfo?buildID=1055922
Build name: "nodejs-6-20180308155546.c2c572ec"
Here, the modulemd [3] lists just one RPM component (SRPM): * nodejs
The page also says "Package Name nodejs" which again links to a "nodejs" package [4] in Koji. But this time, there *is* a "nodejs" package as well as a "nodejs" module. That page lists both!
An example of a "nodejs" package [5], build name "nodejs-9.8.0-1.module_1571+4f4bc63d" which is a good old NVR.
An example of a "nodejs" module [6], build name "nodejs-6-20180308155546.c2c572ec" whic is not an NVR, it's a NSVC (name, stream, version, context).
So after going through these two examples, I think this discussion should be more about: 1. storing two completely different IDs (NVR and NSVC) in the same data structure 2. listing two different build types (package and module) in the same place, both as package builds
[1] https://kojipkgs.fedoraproject.org//packages/golang-ecosystem/2017.0/2018031... [2] https://koji.fedoraproject.org/koji/packageinfo?packageID=25886 [3] https://kojipkgs.fedoraproject.org//packages/nodejs/6/20180308155546.c2c572e... [4] https://koji.fedoraproject.org/koji/packageinfo?packageID=15154 [5] https://koji.fedoraproject.org/koji/buildinfo?buildID=1055873 [6] https://koji.fedoraproject.org/koji/buildinfo?buildID=1055927
On Thu, Mar 15, 2018 at 4:12 PM, Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Thu, Mar 15, 2018 at 03:26:50PM +0100, Patrick Uiterwijk wrote:
As a consequence, when you get an N-S-V.C as modules are represented in Koji builds, doing a .rsplit('-', 2) will not give you Name, Stream, Version.Context per se. You could totally have a module called nodejs-my-stream-5-20170314.abcd, with name=nodejs, stream=my-stream-5, version=20170314, context=abcd. There is no way for you to independently figure out what the NSVC components are, you will need to ask Koji, and use its name, version and release fields (with name=name, version=stream, release=version(.context)).
I really question this change and if we shouldn't fix it as this is going to break every single script trying to get package name from binary/build.
Thanks for all your work Patrick and the heads up about this.
Pierre _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists. fedoraproject.org
infrastructure@lists.fedoraproject.org
-
Adam Samalik -
Patrick Uiterwijk -
Peter Robinson -
Pierre-Yves Chibon