Goodbye nvr.rsplit('-', 2), hello modularity

[Patrick Uiterwijk]

Hi all,

If you maintain any application in Fedora Infra (or outside) that
tries to parse things out of RPM names, you might be interested in
this.

For those wondering where I've been spending most of my time the last
few weeks: I've been deep in the internals of Bodhi fixing all kinds
of issues I've found between it and Modularity (turns out enabling
composing was just the beginning).

At some point, I was informed that for modules, they are throwing all
kinds of old assumptions out of the door.
One of those is the good old Name-Version-Release in RPM land.

Modules do not have an NVR, they have an NSVC: Name, Stream, Version, Context.
Name is the name of the module ("nodejs").
Stream is a stream, like "6", "7" or "8", these are crucial
for the
parallel availability that modules are trying to accomplish.
Version is a date/time stamp, used to indicate new versions in the
same name-stream.
Context is a hash of some information in the built module that makes
it so that the nodejs-6-20170314 built for Fedora 28 has a different
identifier than the one built for Fedora 29.

There are some additional fields for installed things (arch and
profile), but those aren't really important for things trying to
parse/show module names I think.

One of the interesting things they wanted to also allow: dashes in streams.
As a consequence, when you get an N-S-V.C as modules are represented
in Koji builds, doing a .rsplit('-', 2) will not give you Name,
Stream, Version.Context per se.
You could totally have a module called
nodejs-my-stream-5-20170314.abcd, with name=nodejs,
stream=my-stream-5, version=20170314, context=abcd.
There is no way for you to independently figure out what the NSVC
components are, you will need to ask Koji, and use its name, version
and release fields (with name=name, version=stream,
release=version(.context)).

Also as a consequence, modules should not be dash-separated in
anything the users see.
For user consumption, they are Name:Stream:Version:Context, so you may
need to manually convert between one representation and the other if
you need to look up in koji or other systems versus display to users.

Also, note that if you are touching older modules in Koji (e.g.
because you go through all koji builds), modules before February 19th
will be lacking the Context field.
However, I've been told that those modules will not be used, so unless
you or your app goes spelunking in koji (for fun and profit), you
should never see those.

I hope that this is useful information for anyone else finding
themselves having to parse NVRs/NSVs.

Regards,
Patrick


p.s.: This is a forward of my email sent originally to the Infra list.

[Mátyás Selmeci]

On 03/15/2018 10:14 AM, Patrick Uiterwijk wrote:
>  Hi all,
>
>  If you maintain any application in Fedora Infra (or outside) that
>  tries to parse things out of RPM names, you might be interested in
>  this.
>
>  For those wondering where I've been spending most of my time the last
>  few weeks: I've been deep in the internals of Bodhi fixing all kinds
>  of issues I've found between it and Modularity (turns out enabling
>  composing was just the beginning).
>
>  At some point, I was informed that for modules, they are throwing all
>  kinds of old assumptions out of the door.
>  One of those is the good old Name-Version-Release in RPM land.
>
>  Modules do not have an NVR, they have an NSVC: Name, Stream, Version, Context.
>  Name is the name of the module ("nodejs").
>  Stream is a stream, like "6", "7" or "8", these are crucial
> for the
>  parallel availability that modules are trying to accomplish.
>  Version is a date/time stamp, used to indicate new versions in the
>  same name-stream.
>  Context is a hash of some information in the built module that makes
>  it so that the nodejs-6-20170314 built for Fedora 28 has a different
>  identifier than the one built for Fedora 29.
>
>  There are some additional fields for installed things (arch and
>  profile), but those aren't really important for things trying to
>  parse/show module names I think.
>
>  One of the interesting things they wanted to also allow: dashes in streams.
>  As a consequence, when you get an N-S-V.C as modules are represented
>  in Koji builds, doing a .rsplit('-', 2) will not give you Name,
>  Stream, Version.Context per se.
>  You could totally have a module called
>  nodejs-my-stream-5-20170314.abcd, with name=nodejs,
>  stream=my-stream-5, version=20170314, context=abcd.
>  There is no way for you to independently figure out what the NSVC
>  components are, you will need to ask Koji, and use its name, version
>  and release fields (with name=name, version=stream,
>  release=version(.context)).
>
>  Also as a consequence, modules should not be dash-separated in
>  anything the users see.
>  For user consumption, they are Name:Stream:Version:Context, so you may
>  need to manually convert between one representation and the other if
>  you need to look up in koji or other systems versus display to users.
>
>  Also, note that if you are touching older modules in Koji (e.g.
>  because you go through all koji builds), modules before February 19th
>  will be lacking the Context field.
>  However, I've been told that those modules will not be used, so unless
>  you or your app goes spelunking in koji (for fun and profit), you
>  should never see those.
>
>  I hope that this is useful information for anyone else finding
>  themselves having to parse NVRs/NSVs.
>
>  Regards,
>  Patrick 
This does not affect traditional RPM/SRPM packages, right?

Thanks,
-Mat

[Patrick Uiterwijk]

>  This does not affect traditional RPM/SRPM packages, right? 
Correct, this only impacts modules.

>
> Thanks,
> -Mat
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org

[Tom Hughes]

On 15/03/18 15:30, Patrick Uiterwijk wrote:
> >
> >
> > This does not affect traditional RPM/SRPM packages, right?
>  
>  Correct, this only impacts modules. 
Well currently at any rate? What's the long term story?

Only when some module rpms accidentally leaked into rawhide
recently they definitely had the kind of rather unhelpful names
that you described in your post and I got the impression at
the time that it was anticipated they might come to rawhide
for real at some point.

Tom

-- 
Tom Hughes (tom(a)compton.nu)
http://compton.nu/

[Patrick マルタインアンドレアス Uiterwijk]

>  On 15/03/18 15:30, Patrick Uiterwijk wrote:
>  
>  Well currently at any rate? What's the long term story?
>  
>  Only when some module rpms accidentally leaked into rawhide
>  recently they definitely had the kind of rather unhelpful names
>  that you described in your post and I got the impression at
>  the time that it was anticipated they might come to rawhide
>  for real at some point. 
Note that the module builds I was talking about are the actual modules, not the module
content.
The RPMs that ended up in rawhide standard repos (which was a bug, they should have gone
to, and will go to a Modular side-repo), are standard RPMS with NVR like
$packagename-$version-$release.module_$tag.
In other words, the packages that end up in the repos fit the NVR pattern, it's just
that the release part has its .%{dist} replaced with .module_%{tag}.

My email was about module description builds as you'd submit them to Bodhi for
example, like https://koji.fedoraproject.org/koji/buildinfo?buildID=1055963.

> 
> Tom

[Nico Kadel-Garcia]

On Thu, Mar 15, 2018 at 11:14 AM, Patrick Uiterwijk
<puiterwijk(a)redhat.com&gt; wrote:
>  Hi all,
>
>  If you maintain any application in Fedora Infra (or outside) that
>  tries to parse things out of RPM names, you might be interested in
>  this.
>
>  For those wondering where I've been spending most of my time the last
>  few weeks: I've been deep in the internals of Bodhi fixing all kinds
>  of issues I've found between it and Modularity (turns out enabling
>  composing was just the beginning).
>
>  At some point, I was informed that for modules, they are throwing all
>  kinds of old assumptions out of the door.
>  One of those is the good old Name-Version-Release in RPM land.
>
>  Modules do not have an NVR, they have an NSVC: Name, Stream, Version, Context.
>  Name is the name of the module ("nodejs").
>  Stream is a stream, like "6", "7" or "8", these are crucial
> for the
>  parallel availability that modules are trying to accomplish.
>  Version is a date/time stamp, used to indicate new versions in the
>  same name-stream.
>  Context is a hash of some information in the built module that makes
>  it so that the nodejs-6-20170314 built for Fedora 28 has a different
>  identifier than the one built for Fedora 29. 
Isn't that what "%{dist}" is for? Do you have a pointer to where this
is laid out?

We've seen people try to re-invent version numbering before.with other
applications, to add features for their desired workflow. Examples
includes the hashes added to python module versions at pypi.org, which
seriously broke py2pack and forced them to add an impedance matching
utility on their website so the old tools continued to work. Is this a
really desirable change?

[Daniel P. Berrangé]

On Thu, Mar 15, 2018 at 04:14:34PM +0100, Patrick Uiterwijk wrote:
>  Hi all,
>  
>  If you maintain any application in Fedora Infra (or outside) that
>  tries to parse things out of RPM names, you might be interested in
>  this.
>  
>  For those wondering where I've been spending most of my time the last
>  few weeks: I've been deep in the internals of Bodhi fixing all kinds
>  of issues I've found between it and Modularity (turns out enabling
>  composing was just the beginning).
>  
>  At some point, I was informed that for modules, they are throwing all
>  kinds of old assumptions out of the door.
>  One of those is the good old Name-Version-Release in RPM land.
>  
>  Modules do not have an NVR, they have an NSVC: Name, Stream, Version, Context.
>  Name is the name of the module ("nodejs").
>  Stream is a stream, like "6", "7" or "8", these are crucial
> for the
>  parallel availability that modules are trying to accomplish.
>  Version is a date/time stamp, used to indicate new versions in the
>  same name-stream.
>  Context is a hash of some information in the built module that makes
>  it so that the nodejs-6-20170314 built for Fedora 28 has a different
>  identifier than the one built for Fedora 29.
>  
>  There are some additional fields for installed things (arch and
>  profile), but those aren't really important for things trying to
>  parse/show module names I think.
>  
>  One of the interesting things they wanted to also allow: dashes in streams.
>  As a consequence, when you get an N-S-V.C as modules are represented
>  in Koji builds, doing a .rsplit('-', 2) will not give you Name,
>  Stream, Version.Context per se.
>  You could totally have a module called
>  nodejs-my-stream-5-20170314.abcd, with name=nodejs,
>  stream=my-stream-5, version=20170314, context=abcd.
>  There is no way for you to independently figure out what the NSVC
>  components are, you will need to ask Koji, and use its name, version
>  and release fields (with name=name, version=stream,
>  release=version(.context)). 
Have a non-deterministic naming format whose ambiguity you can only
resolve by querying Koji feels like a pretty undesirable thing. At
least it doesn't leak into the RPMs themselves, but from a developer
pov it would be really nice to be able to unambiguously parse the
module names into their parts without calling out to RPC services.
There may be contexts where you want to do such things where you
would rather not have outbound network connectivity to anywhere,
let along Koji.

If we need to allow '-' in the streams, could we use something
other than '-' as the field separator for the module names to
remove ambiguity ?

>  For user consumption, they are Name:Stream:Version:Context, so you
> may
>  need to manually convert between one representation and the other if
>  you need to look up in koji or other systems versus display to users. 
What stops us using Name:Stream:Version:Context everywhere,
or  Name:Stream:Context-Version or Name:Stream-Version-Context
if we need it closer to RPM NVR style ?


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Chuck Anderson]

On Thu, Mar 15, 2018 at 04:19:46PM +0000, Daniel P. Berrangé wrote:
>  On Thu, Mar 15, 2018 at 04:14:34PM +0100, Patrick Uiterwijk wrote:
>  > For user consumption, they are Name:Stream:Version:Context, so you may
>  > need to manually convert between one representation and the other if
>  > you need to look up in koji or other systems versus display to users.
>  
>  What stops us using Name:Stream:Version:Context everywhere,
>  or  Name:Stream:Context-Version or Name:Stream-Version-Context
>  if we need it closer to RPM NVR style ? 
: doesn't work very well in filenames due to it being a pathname separator in some
filesystems among other things.

[Randy Barlow]

On 03/15/2018 12:28 PM, Chuck Anderson wrote:
>  : doesn't work very well in filenames due to it being a pathname
> separator in some filesystems among other things. 
It sounds like this breaks a great deal of tooling. Can we reconsider
switching away from - separators in modules? Is allowing streams to have
-'s in them important enough to break so many tools?

[Simo Sorce]

On Thu, 2018-03-15 at 14:06 -0400, Randy Barlow wrote:
>  On 03/15/2018 12:28 PM, Chuck Anderson wrote:
>  > : doesn't work very well in filenames due to it being a pathname separator in
> some filesystems among other things.
>  
>  It sounds like this breaks a great deal of tooling. Can we reconsider
>  switching away from - separators in modules? Is allowing streams to have
>  -'s in them important enough to break so many tools? 
I vote for using ⨊ as the separator, makes things easier :-)

Simo.

-- 
Simo Sorce
Sr. Principal Software Engineer
Red Hat, Inc

[langdon]

On 03/15/2018 04:31 PM, Simo Sorce wrote:
>  On Thu, 2018-03-15 at 14:06 -0400, Randy Barlow wrote:
> > On 03/15/2018 12:28 PM, Chuck Anderson wrote:
> >> : doesn't work very well in filenames due to it being a pathname separator in
> some filesystems among other things.
> > It sounds like this breaks a great deal of tooling. Can we reconsider
> > switching away from - separators in modules? Is allowing streams to have
> > -'s in them important enough to break so many tools?
>  I vote for using ⨊ as the separator, makes things easier :-)
>
>  Simo.
>  
kidding aside, we are trying to get metadata forced in to the name 
"field"[1] in to proper metadata fields rather than just mangling the 
names. As a result, we end up with more fields and we don't have to 
guess what they are based on conventions.

I also am not sure why we wouldn't want to have hyphens in the names of 
modules. For example, a package for "react-native". We even say in the 
naming guidelines "When naming packages for Fedora, the maintainer must 
use the dash '-' as the delimiter for name parts." [2]. Although, this 
is a bit unclear as to whether they mean "base name" or the overall rpm 
name, from context, I think it is base name.

I could see an argument for using periods as the metadata separator, 
similar to rpms. However, this might cause confusion with what the 
elements of the metadata "mean" if they are not the same as (and in the 
same order as) the elements of an rpm "name".

We chose a colon & slash (for profile names) somewhat based on the 
practice(s) of docker. However, the dnf team made the ultimate choice, 
so their opinion on the subject matters.

Langdon

[1]: what is referred to in the naming guidelines as the "base name" as 
distinct from the full name of the rpm which would include N,V, & R

[2]: 
https://fedoraproject.org/wiki/Packaging:Naming?rd=Packaging:NamingGuidel...

[Adam Williamson]

On Thu, 2018-03-15 at 19:57 -0400, langdon wrote:
>  On 03/15/2018 04:31 PM, Simo Sorce wrote:
>  > On Thu, 2018-03-15 at 14:06 -0400, Randy Barlow wrote:
>  > > On 03/15/2018 12:28 PM, Chuck Anderson wrote:
>  > > > : doesn't work very well in filenames due to it being a pathname
> separator in some filesystems among other things.
>  > > 
>  > > It sounds like this breaks a great deal of tooling. Can we reconsider
>  > > switching away from - separators in modules? Is allowing streams to have
>  > > -'s in them important enough to break so many tools?
>  > 
>  > I vote for using ⨊ as the separator, makes things easier :-)
>  > 
>  > Simo.
>  > 
>  
>  kidding aside, we are trying to get metadata forced in to the name 
>  "field"[1] in to proper metadata fields rather than just mangling the 
>  names. As a result, we end up with more fields and we don't have to 
>  guess what they are based on conventions.
>  
>  I also am not sure why we wouldn't want to have hyphens in the names of 
>  modules. For example, a package for "react-native". 
If you are trying to parse a delimited string, you can cope with either
the *first* or *last* field also being allowed to contain the
delimiter. This is how it works for RPM NEVRAs: name can contain a -,
but epoch, version, release and arch cannot. This allows reliable
parsing because you know any 'extra' -s must be part of the name;
practically speaking, as the subject of the thread suggests, you can
just do a .rsplit() (or equivalent in your language of choice) to get
the fields.

So: there would be no problem with - in the *name* of the modules, so
long as that's the first field, which apparently it is.

The problem is if you allow the delimiter in *both* the first and last
fields, or in *any other field besides the first and the last*. As soon
as you do that, parsing is impossible. This is the issue with module
names, as Patrick pointed out: they allow - in the stream, which is the
*second* field. As soon as you allow this, reliably parsing the name
components out of the whole name becomes impossible.

Yes, you can say 'just don't parse strings, go ask a service instead',
but parsing strings is sometimes by a long way the most convenient way
to do things, and I really can't see how allowing the delimiter
character to appear in one of the field names is a *bigger* win than
allowing the components to be reliably parsed from the overall name.

(I have extensive experience with this field, thanks to dealing with
RPM names, image filenames, *and* Pungi compose IDs (oh God, don't make
me break out my compose ID parser as an example), and at this point I'd
like to designate a special circle of hell for people who do this sort
of thing. :>)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Pierre-Yves Chibon]

On Thu, Mar 15, 2018 at 05:47:41PM -0700, Adam Williamson wrote:
>  So: there would be no problem with - in the *name* of the modules,
> so
>  long as that's the first field, which apparently it is.
>  
>  The problem is if you allow the delimiter in *both* the first and last
>  fields, or in *any other field besides the first and the last*. As soon
>  as you do that, parsing is impossible. This is the issue with module
>  names, as Patrick pointed out: they allow - in the stream, which is the
>  *second* field. As soon as you allow this, reliably parsing the name
>  components out of the whole name becomes impossible.
>  
>  Yes, you can say 'just don't parse strings, go ask a service instead',
>  but parsing strings is sometimes by a long way the most convenient way
>  to do things, and I really can't see how allowing the delimiter
>  character to appear in one of the field names is a *bigger* win than
>  allowing the components to be reliably parsed from the overall name. 
I wonder what we can do about this. Is it FESCo material?
Can the folks working on modularity comment more on this?

This is quite failing the moto: it's ok to disagree, not to surprise


Pierre

[Randy Barlow]

On 03/16/2018 05:18 AM, Pierre-Yves Chibon wrote:
>  I wonder what we can do about this. Is it FESCo material?
>  Can the folks working on modularity comment more on this? 
I am considering filing it for FESCo to consider. I haven't seen a
compelling explanation for why modules are using a different separator
than our other content types so far, and it really does break a lot of
tooling.

Koji also does not allow the :'s (so modules continue to use -'s there),
so it also causes an inconsistency in NSVC syntax for Koji and
everything else (which will be very confusing in the Bodhi UI, which
touches "everything else" as well as Koji).

[Dennis Gregorovic]

modules are not RPMs.  I would not expect them to necessarily use the same
format as RPMs.  If we take koji out of the equation, we have module builds
in N:S:V:C format and module RPMs in N-V-R.A format.  They use different
separators, but both can be parsed consistently.

The challenge here is that those module builds are squeezed into a N-V-R
format when imported into Koji.  The N-V-R format is used for all build
types in Koji and it works well for its purpose, but it's also not
realistic to expect that all content types are going to natively use N-V-R
outside of Koji. My suggestion is that we consider the N-V-R format of
modules to be a representation internal to koji and that N:S:V:C is the
format used to represent modules to users.  If that means tweaking the Koji
UI we can look into that.

Cheers
-- Dennis



On Fri, Mar 16, 2018 at 3:40 PM, Randy Barlow <bowlofeggs(a)fedoraproject.org&gt;
wrote:

>  On 03/16/2018 05:18 AM, Pierre-Yves Chibon wrote:
>  > I wonder what we can do about this. Is it FESCo material?
>  > Can the folks working on modularity comment more on this?
>
>  I am considering filing it for FESCo to consider. I haven't seen a
>  compelling explanation for why modules are using a different separator
>  than our other content types so far, and it really does break a lot of
>  tooling.
>
>  Koji also does not allow the :'s (so modules continue to use -'s there),
>  so it also causes an inconsistency in NSVC syntax for Koji and
>  everything else (which will be very confusing in the Bodhi UI, which
>  touches "everything else" as well as Koji).
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  


-- 
Dennis Gregorovic
Manager, PnT DevOps
Red Hat
dgregor(a)redhat.com    T: +1-978.392.3112    M: +1-617.901.9799

[Randy Barlow]

On 03/16/2018 04:04 PM, Dennis Gregorovic wrote:
>  modules are not RPMs.  I would not expect them to necessarily use
> the
>  same format as RPMs.  If we take koji out of the equation, we have
>  module builds in N:S:V:C format and module RPMs in N-V-R.A format.  They
>  use different separators, but both can be parsed consistently.  
I don't contest the above, but I am arguing that it is needlessly
different and causes infrastructure software to need to handle module
strings differently than they handle RPMs and containers. Perhaps there
is a benefit that I've not been made aware of yet, but from where I sit
it seems like a change that causes problems without bringing a tangible
benefit.

The reasoning I've heard so far is that this allows stream names to have
-'s in them - is that important? I don't know it to be, but am open to
be convinced. Thus my question - is it important enough to have -'s in
stream names to justify the work needed to make all things that interact
with Koji parse them using a web service rather than local code (such as
rsplit('-', 2) in Python)?

>  If that means
>  tweaking the Koji UI we can look into that. 
For my concerns, it would be more helpful to tweak the Koji API than the
UI, so that tools that interact with Koji (like Bodhi) don't have to
know that Modules are different in Koji than they are everywhere else.
Of course, I'd rather just having modules not use -'s in stream names as
I wrote above, but if we really are going that route it'd be ideal if we
used :'s everywhere, including Koji.

[Adam Williamson]

On Fri, 2018-03-16 at 16:14 -0400, Randy Barlow wrote:
>  
>  > If that means
>  > tweaking the Koji UI we can look into that.
>  
>  For my concerns, it would be more helpful to tweak the Koji API than the
>  UI, so that tools that interact with Koji (like Bodhi) don't have to
>  know that Modules are different in Koji than they are everywhere else.
>  Of course, I'd rather just having modules not use -'s in stream names as
>  I wrote above, but if we really are going that route it'd be ideal if we
>  used :'s everywhere, including Koji. 
Thanks, that's what I was trying to get at. The problem is that Koji is
very big and very hairy and no-one enjoys poking it much.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Dennis Gregorovic]

Don't be afraid of koji. :)

It is under active development.  Of course, resources are limited and not
everything can be taken on, but it's worth having the discussion.  I have
filed https://pagure.io/koji/issue/851 as a strawman and will try to
solicit feedback on the idea.

Patrick, if https://pagure.io/koji/issue/851 was implemented would that
resolve the challenge you were facing?  If not, it might be worth reaching
out to koji-devel(a)lists.fedorahosted.org and discussing it further there.

Cheers
-- Dennis

On Fri, Mar 16, 2018 at 6:21 PM, Adam Williamson <adamwill(a)fedoraproject.org
>  wrote: 
>  On Fri, 2018-03-16 at 16:14 -0400, Randy Barlow wrote:
>  >
>  > > If that means
>  > > tweaking the Koji UI we can look into that.
>  >
>  > For my concerns, it would be more helpful to tweak the Koji API than the
>  > UI, so that tools that interact with Koji (like Bodhi) don't have to
>  > know that Modules are different in Koji than they are everywhere else.
>  > Of course, I'd rather just having modules not use -'s in stream names as
>  > I wrote above, but if we really are going that route it'd be ideal if we
>  > used :'s everywhere, including Koji.
>
>  Thanks, that's what I was trying to get at. The problem is that Koji is
>  very big and very hairy and no-one enjoys poking it much.
>  --
>  Adam Williamson
>  Fedora QA Community Monkey
>  IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
>  http://www.happyassassin.net
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  


-- 
Dennis Gregorovic
Manager, PnT DevOps
Red Hat
dgregor(a)redhat.com    T: +1-978.392.3112    M: +1-617.901.9799

[Patrick マルタインアンドレアス Uiterwijk]

>  Don't be afraid of koji. :) 
I couldn't agree more, Koji's a pretty nice project to work on.
It might seems scary for newcomers because of all the ingrained knowledge and assumptions,
but once you get close the community is nice and friendly :).

>  
>  It is under active development.  Of course, resources are limited and not
>  everything can be taken on, but it's worth having the discussion.  I have
>  filed https://pagure.io/koji/issue/851 as a strawman and will try to
>  solicit feedback on the idea.
>   
Thanks for doing that, though I don't know that it would resolve my challenges.
First, I'd like to point out that by now I've basically solved the problems for
Bodhi and know how to deal with this change.
The main reason I wanted to send this email is because in things like fedmsg's and
other things, the NVRs as imported by Koji still leak out.
Even if the identifiers are colon-separated in all places and the N-S-V encoded version is
never exported out of koji, people will still need to update scripts and tooling, and
bringing that to people's attention was my main reason for sending the email.

I absolutely did not mean for my email to be judgemental, as I've not been privy to
the decision to allow dashes in multiple fields or the rest, so I can't really judge
the reasoning or the conclusion.
If it came over as judging, I'm sorry, I really just wanted to give people a headsup
so that they have some upfront time to update scripts before seeing eithe module
identifiers in fedmsg's or any other place.

>  Patrick, if https://pagure.io/koji/issue/851 was implemented would
> that
>  resolve the challenge you were facing?  If not, it might be worth reaching
>  out to koji-devel(a)lists.fedorahosted.org and discussing it further there. 
It would not, but that's okay.
It would still mean that Module identifiers, whether you notice them as N-S-V.C or
N:S:V:C, are new, and all tooling and scripts will need to be updated to handle those new
identifiers.

>  
>  Cheers
>  -- Dennis 
Patrick

[Adam Williamson]

On Sat, 2018-03-17 at 15:15 +0000, Patrick  マルタインアンドレアス  Uiterwijk
wrote:
>  > Don't be afraid of koji. :)
>  
>  I couldn't agree more, Koji's a pretty nice project to work on.
>  It might seems scary for newcomers because of all the ingrained
>  knowledge and assumptions, but once you get close the community is
>  nice and friendly :). 
Sorry, I should've been clearer - I wasn't referring to any of these
attributes, but just to how important Koji is and how many different
key processes it involves. This makes it inherently complex to change
significant things in it because there are just so many other systems
and processes hooked into Koji and it's very difficult to know how
they're all going to react.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Jan Kaluza]

On Sat, Mar 17, 2018 at 4:15 PM, Patrick マルタインアンドレアス Uiterwijk <
puiterwijk(a)redhat.com&gt; wrote:

>  > Don't be afraid of koji. :)
>
>  I couldn't agree more, Koji's a pretty nice project to work on.
>  It might seems scary for newcomers because of all the ingrained knowledge
>  and assumptions, but once you get close the community is nice and friendly
>  :).
>
>  >
>  > It is under active development.  Of course, resources are limited and not
>  > everything can be taken on, but it's worth having the discussion.  I have
>  > filed https://pagure.io/koji/issue/851 as a strawman and will try to
>  > solicit feedback on the idea.
>  >
>
>  Thanks for doing that, though I don't know that it would resolve my
>  challenges.
>  First, I'd like to point out that by now I've basically solved the
>  problems for Bodhi and know how to deal with this change.
>  The main reason I wanted to send this email is because in things like
>  fedmsg's and other things, the NVRs as imported by Koji still leak out.
>  Even if the identifiers are colon-separated in all places and the N-S-V
>  encoded version is never exported out of koji, people will still need to
>  update scripts and tooling, and bringing that to people's attention was my
>  main reason for sending the email.
>
>  I absolutely did not mean for my email to be judgemental, as I've not been
>  privy to the decision to allow dashes in multiple fields or the rest, so I
>  can't really judge the reasoning or the conclusion.
>  If it came over as judging, I'm sorry, I really just wanted to give people
>  a headsup so that they have some upfront time to update scripts before
>  seeing eithe module identifiers in fedmsg's or any other place.
>
>  > Patrick, if https://pagure.io/koji/issue/851 was implemented would that
>  > resolve the challenge you were facing?  If not, it might be worth
>  reaching
>  > out to koji-devel(a)lists.fedorahosted.org and discussing it further
>  there.
>
>  It would not, but that's okay.
>  It would still mean that Module identifiers, whether you notice them as
>  N-S-V.C or N:S:V:C, are new, and all tooling and scripts will need to be
>  updated to handle those new identifiers.
>  
I've read some parts of this thread and I see people are kind of saying "we
have to update all the tooling". I really don't think there is tooling
which needs to be updated *just* because of the delimiter.

If something does not support modules and it should support modules and
*needs* to work with Koji content-generator imported builds for some
reason, it needs to updated to support modules and as part of this update,
it can be told to use ':' as its output.

If something does support modules already, it needs to be updated to
support "context" field anyway and as part of this update ... ^

Note that this is just about the "module" type koji builds imported to Koji
by MBS once the module is built. I think that t if you have no idea that
those builds exist, you have nothing to update in your tool.

What is impacted by this is:
- Bodhi - which is easy to update and I've sent a PR which has been
continued on by Patrick and is merged now.
- Pungi - this supports ":" already.
- MBS, pdc-updater - supports ":" already.

Is there actually anything else?

Regards,
Jan Kaluza


>  >
>  > Cheers
>  > -- Dennis
>
>  Patrick
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>

[Pierre-Yves Chibon]

On Mon, Mar 19, 2018 at 10:10:23AM +0100, Jan Kaluza wrote:
>     If something does not support modules and it should support
> modules and
>     *needs* to work with Koji content-generator imported builds for some
>     reason, it needs to updated to support modules and as part of this update,
>     it can be told to use ':' as its output.
>  
>     If something does support modules already, it needs to be updated to
>     support "context" field anyway and as part of this update ... ^
>  
>     Note that this is just about the "module" type koji builds imported to
>     Koji by MBS once the module is built. I think that t if you have no idea
>     that those builds exist, you have nothing to update in your tool.
>  
>     What is impacted by this is:
>     - Bodhi - which is easy to update and I've sent a PR which has been
>     continued on by Patrick and is merged now.
>     - Pungi - this supports ":" already.
>     - MBS, pdc-updater - supports ":" already.
>     Is there actually anything else? 
This is precisely what scared me a little bit with this change. We have a very
limited knowledge of what it will break.
You noted here the major things it impacts, I would not be entirely surprised
if there would be more we're missing atm and there are definitively high chances
that there are tooling out there that we do not control that is going to break.

That things break is one thing, that Patrick is the one announcing it because he
learned about this the hard way while trying to make a crucial infra tool work
with module is something different (and I'm not even going to ask the question
of why Patrick is the one working on this).


Pierre

[Zbigniew Jędrzejewski-Szmek]

On Fri, Mar 16, 2018 at 04:14:27PM -0400, Randy Barlow wrote:
>  On 03/16/2018 04:04 PM, Dennis Gregorovic wrote:
>  > modules are not RPMs.  I would not expect them to necessarily use the
>  > same format as RPMs.  If we take koji out of the equation, we have
>  > module builds in N:S:V:C format and module RPMs in N-V-R.A format.  They
>  > use different separators, but both can be parsed consistently. 
>  
>  I don't contest the above, but I am arguing that it is needlessly
>  different and causes infrastructure software to need to handle module
>  strings differently than they handle RPMs and containers. Perhaps there
>  is a benefit that I've not been made aware of yet, but from where I sit
>  it seems like a change that causes problems without bringing a tangible
>  benefit.
>  
>  The reasoning I've heard so far is that this allows stream names to have
>  -'s in them - is that important? I don't know it to be, but am open to
>  be convinced. Thus my question - is it important enough to have -'s in
>  stream names to justify the work needed to make all things that interact
>  with Koji parse them using a web service rather than local code (such as
>  rsplit('-', 2) in Python)? 
So, is anyone knowledgeable who can answer this question?

If nobody can, then mostly likely the answer is "no, it's not important".

Zbyszek

[Petr Šabata]

On Sun, Mar 18, 2018 at 07:09:23PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  On Fri, Mar 16, 2018 at 04:14:27PM -0400, Randy Barlow wrote:
>  > On 03/16/2018 04:04 PM, Dennis Gregorovic wrote:
>  > > modules are not RPMs.  I would not expect them to necessarily use the
>  > > same format as RPMs.  If we take koji out of the equation, we have
>  > > module builds in N:S:V:C format and module RPMs in N-V-R.A format.  They
>  > > use different separators, but both can be parsed consistently. 
>  > 
>  > I don't contest the above, but I am arguing that it is needlessly
>  > different and causes infrastructure software to need to handle module
>  > strings differently than they handle RPMs and containers. Perhaps there
>  > is a benefit that I've not been made aware of yet, but from where I sit
>  > it seems like a change that causes problems without bringing a tangible
>  > benefit.
>  > 
>  > The reasoning I've heard so far is that this allows stream names to have
>  > -'s in them - is that important? I don't know it to be, but am open to
>  > be convinced. Thus my question - is it important enough to have -'s in
>  > stream names to justify the work needed to make all things that interact
>  > with Koji parse them using a web service rather than local code (such as
>  > rsplit('-', 2) in Python)?
>  
>  So, is anyone knowledgeable who can answer this question?
>  
>  If nobody can, then mostly likely the answer is "no, it's not important".
The reason for allowing hyphens was to, maybe ironically, allow
more flexibility and to avoid unnecessary restriction on names.

While streams are often presented as upstream versions in
most of the examples, they're strings and are, in reality, an
extension to the name.  Your stream might be just "6" or "8" but
it could also be "experimental-build" or "performance-patches".
Current tooling shortcomings shouldn't influence how details
like this are designed.  The tooling can be fixed (or enhanced,
if you prefer).

The ordering of the fields is based on the importance of
the field to the user and most of them are optional when
people interact with modules using their package manager.
"module", "module:stream", "module:stream:version" are all
valid inputs (we do not expect people to use context directly;
it's just serialized that distinguishes individual builds for
stream expansion).  Context is indeed similar to dist-tag,
just richer.  Given the parallel availability, your modules
aren't built just for a release of Fedora, they are built for
any valid combination of their modular dependncies, with the
Fedora release being just one of them.

Note, as Patrick's already said, these names do not leak
anywhere.  Modules are not RPMs and are not handled as such.
This, AFAIK, only affects what you see in some parts of the
build and update system (and on the bus).

Also note that generally if you have access to modules, you
have access to their (structured) metadata.

P

[Adam Samalik]

(a repost from the infra list)


Please note that modules (and therefore module builds) may reference to
multiple RPM builds. Also, the module name doesn't need to match any RPM
name. Therefore getting package names from the module build name would not
be possible anyway.


As an exapmple, see the following build: https://koji.fedoraproject.
org/koji/buildinfo?buildID=1057280

Build name: "golang-ecosystem-2017.0-20180312141905.40a0ab3e"

If you open the modulemd [1] (a module definition file) linked at the
bottom of the page, you'll see that this module build contains four RPM
components (SRPMs):
 * golang-github-russross-blackfriday
 * go-srpm-macros
 * golang-github-cpuguy83-go-md2man
 * go-compilers

The page also says "Package Name    golang-ecosystem" which links to a
"golang-ecosystem" package [2] in Koji. That's weird, there is no such
package named "golang-ecosystem". The page lists two module builds, not two
package builds.


Another example, where a module references only one RPM package and the
module name matches the package name: https://koji.fedoraproject.
org/koji/buildinfo?buildID=1055922

Build name: "nodejs-6-20180308155546.c2c572ec"

Here, the modulemd [3] lists just one RPM component (SRPM):
 * nodejs

The page also says "Package Name    nodejs" which again links to a
"nodejs"
package [4] in Koji. But this time, there *is* a "nodejs" package as well
as a "nodejs" module. That page lists both!

An example of a "nodejs" package [5], build name
"nodejs-9.8.0-1.module_1571+4f4bc63d" which is a good old NVR.

An example of a "nodejs" module [6], build name
"nodejs-6-20180308155546.c2c572ec"
whic is not an NVR, it's a NSVC (name, stream, version, context).


So after going through these two examples, I think this discussion should
be more about:
 1. storing two completely different IDs (NVR and NSVC) in the same data
structure
 2. listing two different build types (package and module) in the same
place, both as package builds



[1] https://kojipkgs.fedoraproject.org//packages/golang-ecosystem/2017.0/
20180312141905.40a0ab3e/files/module/modulemd.txt
[2] https://koji.fedoraproject.org/koji/packageinfo?packageID=25886
[3] https://kojipkgs.fedoraproject.org//packages/nodejs/6/20180308155546.
c2c572ec/files/module/modulemd.txt
[4] https://koji.fedoraproject.org/koji/packageinfo?packageID=15154
[5] https://koji.fedoraproject.org/koji/buildinfo?buildID=1055873
[6] https://koji.fedoraproject.org/koji/buildinfo?buildID=1055927


On Mon, Mar 19, 2018 at 9:27 AM, Petr Šabata <contyk(a)redhat.com&gt; wrote:

>  On Sun, Mar 18, 2018 at 07:09:23PM +0000, Zbigniew
> Jędrzejewski-Szmek
>  wrote:
>  > On Fri, Mar 16, 2018 at 04:14:27PM -0400, Randy Barlow wrote:
>  > > On 03/16/2018 04:04 PM, Dennis Gregorovic wrote:
>  > > > modules are not RPMs.  I would not expect them to necessarily use the
>  > > > same format as RPMs.  If we take koji out of the equation, we have
>  > > > module builds in N:S:V:C format and module RPMs in N-V-R.A format.
>  They
>  > > > use different separators, but both can be parsed consistently.
>  > >
>  > > I don't contest the above, but I am arguing that it is needlessly
>  > > different and causes infrastructure software to need to handle module
>  > > strings differently than they handle RPMs and containers. Perhaps there
>  > > is a benefit that I've not been made aware of yet, but from where I sit
>  > > it seems like a change that causes problems without bringing a tangible
>  > > benefit.
>  > >
>  > > The reasoning I've heard so far is that this allows stream names to
>  have
>  > > -'s in them - is that important? I don't know it to be, but am open to
>  > > be convinced. Thus my question - is it important enough to have -'s in
>  > > stream names to justify the work needed to make all things that
>  interact
>  > > with Koji parse them using a web service rather than local code (such
>  as
>  > > rsplit('-', 2) in Python)?
>  >
>  > So, is anyone knowledgeable who can answer this question?
>  >
>  > If nobody can, then mostly likely the answer is "no, it's not
> important".
>
>  The reason for allowing hyphens was to, maybe ironically, allow
>  more flexibility and to avoid unnecessary restriction on names.
>
>  While streams are often presented as upstream versions in
>  most of the examples, they're strings and are, in reality, an
>  extension to the name.  Your stream might be just "6" or "8" but
>  it could also be "experimental-build" or "performance-patches".
>  Current tooling shortcomings shouldn't influence how details
>  like this are designed.  The tooling can be fixed (or enhanced,
>  if you prefer).
>
>  The ordering of the fields is based on the importance of
>  the field to the user and most of them are optional when
>  people interact with modules using their package manager.
>  "module", "module:stream", "module:stream:version" are all
>  valid inputs (we do not expect people to use context directly;
>  it's just serialized that distinguishes individual builds for
>  stream expansion).  Context is indeed similar to dist-tag,
>  just richer.  Given the parallel availability, your modules
>  aren't built just for a release of Fedora, they are built for
>  any valid combination of their modular dependncies, with the
>  Fedora release being just one of them.
>
>  Note, as Patrick's already said, these names do not leak
>  anywhere.  Modules are not RPMs and are not handled as such.
>  This, AFAIK, only affects what you see in some parts of the
>  build and update system (and on the bus).
>
>  Also note that generally if you have access to modules, you
>  have access to their (structured) metadata.
>
>  P
>
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>
>  

-- 

Adam Šamalík
---------------------------
Software Engineer
Red Hat

[Zbigniew Jędrzejewski-Szmek]

On Mon, Mar 19, 2018 at 09:27:30AM +0100, Petr Šabata wrote:
>  On Sun, Mar 18, 2018 at 07:09:23PM +0000, Zbigniew Jędrzejewski-Szmek
> wrote:
>  > On Fri, Mar 16, 2018 at 04:14:27PM -0400, Randy Barlow wrote:
>  > > On 03/16/2018 04:04 PM, Dennis Gregorovic wrote:
>  > > > modules are not RPMs.  I would not expect them to necessarily use the
>  > > > same format as RPMs.  If we take koji out of the equation, we have
>  > > > module builds in N:S:V:C format and module RPMs in N-V-R.A format.  They
>  > > > use different separators, but both can be parsed consistently. 
>  > > 
>  > > I don't contest the above, but I am arguing that it is needlessly
>  > > different and causes infrastructure software to need to handle module
>  > > strings differently than they handle RPMs and containers. Perhaps there
>  > > is a benefit that I've not been made aware of yet, but from where I sit
>  > > it seems like a change that causes problems without bringing a tangible
>  > > benefit.
>  > > 
>  > > The reasoning I've heard so far is that this allows stream names to have
>  > > -'s in them - is that important? I don't know it to be, but am open to
>  > > be convinced. Thus my question - is it important enough to have -'s in
>  > > stream names to justify the work needed to make all things that interact
>  > > with Koji parse them using a web service rather than local code (such as
>  > > rsplit('-', 2) in Python)?
>  > 
>  > So, is anyone knowledgeable who can answer this question?
>  > 
>  > If nobody can, then mostly likely the answer is "no, it's not
> important".
>  
>  The reason for allowing hyphens was to, maybe ironically, allow
>  more flexibility and to avoid unnecessary restriction on names.
>
>  While streams are often presented as upstream versions in
>  most of the examples, they're strings and are, in reality, an
>  extension to the name.  Your stream might be just "6" or "8" but
>  it could also be "experimental-build" or "performance-patches".
>  Current tooling shortcomings shouldn't influence how details
>  like this are designed.  The tooling can be fixed (or enhanced,
>  if you prefer). 
Thank you for the detailed reply.

I see the reasons, and the elegance of not placing restrictions on the
stream field, but I think that's one of those where practicality beats
purity. _Tooling_ may be improved, but this is also about _humans_.
With the current format, it is simply impossible to look at a module
n-s-v-c string and parse it into components. In other words, it's not
possible to say what the *name* is. This is a major practical issue.

Frankly, I don't think it's a big restriction to forbid dashes in
stream name. Colons, semicolons, slashes, spaces, and a host of other
characters have to be forbidden too. If we establish a convention to
use plus, underscore, dot, or whatever is the most appropriate, in
a few months people will not remember why they ever wanted dashes.

Zbyszek

>  The ordering of the fields is based on the importance of
>  the field to the user and most of them are optional when
>  people interact with modules using their package manager.
>  "module", "module:stream", "module:stream:version" are all
>  valid inputs (we do not expect people to use context directly;
>  it's just serialized that distinguishes individual builds for
>  stream expansion).  Context is indeed similar to dist-tag,
>  just richer.  Given the parallel availability, your modules
>  aren't built just for a release of Fedora, they are built for
>  any valid combination of their modular dependncies, with the
>  Fedora release being just one of them.
>  
>  Note, as Patrick's already said, these names do not leak
>  anywhere.  Modules are not RPMs and are not handled as such.
>  This, AFAIK, only affects what you see in some parts of the
>  build and update system (and on the bus).
>  
>  Also note that generally if you have access to modules, you
>  have access to their (structured) metadata.

[Petr Šabata]

On Mon, Mar 19, 2018 at 08:57:08AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  On Mon, Mar 19, 2018 at 09:27:30AM +0100, Petr Šabata wrote:
>  > On Sun, Mar 18, 2018 at 07:09:23PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > > On Fri, Mar 16, 2018 at 04:14:27PM -0400, Randy Barlow wrote:
>  > > > On 03/16/2018 04:04 PM, Dennis Gregorovic wrote:
>  > > > > modules are not RPMs.  I would not expect them to necessarily use
> the
>  > > > > same format as RPMs.  If we take koji out of the equation, we have
>  > > > > module builds in N:S:V:C format and module RPMs in N-V-R.A format. 
> They
>  > > > > use different separators, but both can be parsed consistently. 
>  > > > 
>  > > > I don't contest the above, but I am arguing that it is needlessly
>  > > > different and causes infrastructure software to need to handle module
>  > > > strings differently than they handle RPMs and containers. Perhaps there
>  > > > is a benefit that I've not been made aware of yet, but from where I
> sit
>  > > > it seems like a change that causes problems without bringing a tangible
>  > > > benefit.
>  > > > 
>  > > > The reasoning I've heard so far is that this allows stream names to
> have
>  > > > -'s in them - is that important? I don't know it to be, but am
> open to
>  > > > be convinced. Thus my question - is it important enough to have -'s
> in
>  > > > stream names to justify the work needed to make all things that interact
>  > > > with Koji parse them using a web service rather than local code (such as
>  > > > rsplit('-', 2) in Python)?
>  > > 
>  > > So, is anyone knowledgeable who can answer this question?
>  > > 
>  > > If nobody can, then mostly likely the answer is "no, it's not
> important".
>  > 
>  > The reason for allowing hyphens was to, maybe ironically, allow
>  > more flexibility and to avoid unnecessary restriction on names.
>  >
>  > While streams are often presented as upstream versions in
>  > most of the examples, they're strings and are, in reality, an
>  > extension to the name.  Your stream might be just "6" or "8"
> but
>  > it could also be "experimental-build" or "performance-patches".
>  > Current tooling shortcomings shouldn't influence how details
>  > like this are designed.  The tooling can be fixed (or enhanced,
>  > if you prefer).
>  Thank you for the detailed reply.
>  
>  I see the reasons, and the elegance of not placing restrictions on the
>  stream field, but I think that's one of those where practicality beats
>  purity. _Tooling_ may be improved, but this is also about _humans_.
>  With the current format, it is simply impossible to look at a module
>  n-s-v-c string and parse it into components. In other words, it's not
>  possible to say what the *name* is. This is a major practical issue. 
We would really prefer people looking at the structured data
rather than the imperfect string when processing modules.
Modules carry quite a lot of it; strings are just identifiers.
I know it's tempting but doing so might produce suboptimal
results.  That said, if people here (or FESCo) feel like decoding
ID strings is crucial, another policy limiting the options in
Fedora might always be added.  I'd say it'd be unfortunate, though.

>  Frankly, I don't think it's a big restriction to forbid
> dashes in
>  stream name. Colons, semicolons, slashes, spaces, and a host of other
>  characters have to be forbidden too. If we establish a convention to
>  use plus, underscore, dot, or whatever is the most appropriate, in
>  a few months people will not remember why they ever wanted dashes. 
Yes, but unlike all of the examples above, dashes are
comparatively more common as word delimeters in this context and
I think is the most natural choice.  New rules and conventions
can be defined, of course, but it's still forcing people to
work around ambiguous technological limitations.

P

>  
>  Zbyszek
>  
>  > The ordering of the fields is based on the importance of
>  > the field to the user and most of them are optional when
>  > people interact with modules using their package manager.
>  > "module", "module:stream", "module:stream:version" are
> all
>  > valid inputs (we do not expect people to use context directly;
>  > it's just serialized that distinguishes individual builds for
>  > stream expansion).  Context is indeed similar to dist-tag,
>  > just richer.  Given the parallel availability, your modules
>  > aren't built just for a release of Fedora, they are built for
>  > any valid combination of their modular dependncies, with the
>  > Fedora release being just one of them.
>  > 
>  > Note, as Patrick's already said, these names do not leak
>  > anywhere.  Modules are not RPMs and are not handled as such.
>  > This, AFAIK, only affects what you see in some parts of the
>  > build and update system (and on the bus).
>  > 
>  > Also note that generally if you have access to modules, you
>  > have access to their (structured) metadata.
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org

[Fabio Valentini]

On Mon, Mar 19, 2018, 10:20 Petr Šabata <contyk(a)redhat.com&gt; wrote:

>  On Mon, Mar 19, 2018 at 08:57:08AM +0000, Zbigniew
> Jędrzejewski-Szmek
>  wrote:
>  > On Mon, Mar 19, 2018 at 09:27:30AM +0100, Petr Šabata wrote:
>  > > On Sun, Mar 18, 2018 at 07:09:23PM +0000, Zbigniew Jędrzejewski-Szmek
>  wrote:
>  > > > On Fri, Mar 16, 2018 at 04:14:27PM -0400, Randy Barlow wrote:
>  > > > > On 03/16/2018 04:04 PM, Dennis Gregorovic wrote:
>  > > > > > modules are not RPMs.  I would not expect them to necessarily
>  use the
>  > > > > > same format as RPMs.  If we take koji out of the equation, we
>  have
>  > > > > > module builds in N:S:V:C format and module RPMs in N-V-R.A
>  format.  They
>  > > > > > use different separators, but both can be parsed consistently.
>  > > > >
>  > > > > I don't contest the above, but I am arguing that it is
> needlessly
>  > > > > different and causes infrastructure software to need to handle
>  module
>  > > > > strings differently than they handle RPMs and containers. Perhaps
>  there
>  > > > > is a benefit that I've not been made aware of yet, but from
> where
>  I sit
>  > > > > it seems like a change that causes problems without bringing a
>  tangible
>  > > > > benefit.
>  > > > >
>  > > > > The reasoning I've heard so far is that this allows stream names
>  to have
>  > > > > -'s in them - is that important? I don't know it to be, but
> am
>  open to
>  > > > > be convinced. Thus my question - is it important enough to have
>  -'s in
>  > > > > stream names to justify the work needed to make all things that
>  interact
>  > > > > with Koji parse them using a web service rather than local code
>  (such as
>  > > > > rsplit('-', 2) in Python)?
>  > > >
>  > > > So, is anyone knowledgeable who can answer this question?
>  > > >
>  > > > If nobody can, then mostly likely the answer is "no, it's not
>  important".
>  > >
>  > > The reason for allowing hyphens was to, maybe ironically, allow
>  > > more flexibility and to avoid unnecessary restriction on names.
>  > >
>  > > While streams are often presented as upstream versions in
>  > > most of the examples, they're strings and are, in reality, an
>  > > extension to the name.  Your stream might be just "6" or
> "8" but
>  > > it could also be "experimental-build" or
> "performance-patches".
>  > > Current tooling shortcomings shouldn't influence how details
>  > > like this are designed.  The tooling can be fixed (or enhanced,
>  > > if you prefer).
>  > Thank you for the detailed reply.
>  >
>  > I see the reasons, and the elegance of not placing restrictions on the
>  > stream field, but I think that's one of those where practicality beats
>  > purity. _Tooling_ may be improved, but this is also about _humans_.
>  > With the current format, it is simply impossible to look at a module
>  > n-s-v-c string and parse it into components. In other words, it's not
>  > possible to say what the *name* is. This is a major practical issue.
>
>  We would really prefer people looking at the structured data
>  rather than the imperfect string when processing modules.
>  Modules carry quite a lot of it; strings are just identifiers.
>  I know it's tempting but doing so might produce suboptimal
>  results.  That said, if people here (or FESCo) feel like decoding
>  ID strings is crucial, another policy limiting the options in
>  Fedora might always be added.  I'd say it'd be unfortunate, though.
>
>  > Frankly, I don't think it's a big restriction to forbid dashes in
>  > stream name. Colons, semicolons, slashes, spaces, and a host of other
>  > characters have to be forbidden too. If we establish a convention to
>  > use plus, underscore, dot, or whatever is the most appropriate, in
>  > a few months people will not remember why they ever wanted dashes.
>
>  Yes, but unlike all of the examples above, dashes are
>  comparatively more common as word delimeters in this context and
>  I think is the most natural choice.  New rules and conventions
>  can be defined, of course, but it's still forcing people to
>  work around ambiguous technological limitations.
>
>  P
>  
So you think having to send a request to a web service instead of just
parsing a string locally with one line of code is a good trade-off for
allowing dashes?

Gaining one additional allowed character and therefore losing the ability
to parse this locally just seems mad to me ... compared to that,
introducing a "convention" of using underscores (or sth. else) as word
delimiters within fields looks more reasonable by multiple orders of
magnitude ...

Just my 2c.

Fabio

>  Zbyszek
>  >
>  > > The ordering of the fields is based on the importance of
>  > > the field to the user and most of them are optional when
>  > > people interact with modules using their package manager.
>  > > "module", "module:stream",
> "module:stream:version" are all
>  > > valid inputs (we do not expect people to use context directly;
>  > > it's just serialized that distinguishes individual builds for
>  > > stream expansion).  Context is indeed similar to dist-tag,
>  > > just richer.  Given the parallel availability, your modules
>  > > aren't built just for a release of Fedora, they are built for
>  > > any valid combination of their modular dependncies, with the
>  > > Fedora release being just one of them.
>  > >
>  > > Note, as Patrick's already said, these names do not leak
>  > > anywhere.  Modules are not RPMs and are not handled as such.
>  > > This, AFAIK, only affects what you see in some parts of the
>  > > build and update system (and on the bus).
>  > >
>  > > Also note that generally if you have access to modules, you
>  > > have access to their (structured) metadata.
>  > _______________________________________________
>  > devel mailing list -- devel(a)lists.fedoraproject.org
>  > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>

[Stanislav Ochotnicky]

On Mon 19 Mar 2018 09:38:58 AM GMT Fabio Valentini wrote:
>  So you think having to send a request to a web service instead of
> just
>  parsing a string locally with one line of code is a good trade-off for
>  allowing dashes? 
This has been mentioned several times in this thread and I think there's
a misunderstanding around this.

So: When your tool/whatever works with modules it will have to have
module metadata available in some form. In most client-side tools, I'd
guess that will be modulemd metadata in dnf repositories that will get
synced locally (just like rpm/yum repodata). Or it really might be
you'll query koji for the metadata if needed on a system without local
dnf metadata. But if you're working on top of module repositories -
metadata will be there.

RPMs *in* modules still use "N-V-R.A" strings and you can keep parsing
them. That is not changing. But the module that contains them uses
different strategy that allows maintainers more flexibility. And there
does not have to be exact match between module/stream etc name and NVRs
of rpms inside as others have mentioned.


-- 
Stanislav Ochotnicky <sochotnicky(a)redhat.com&gt;
Software Engineer, PnT DevOps - Brno

PGP: F434 2286 27DC 7D9B 2B64  0866 BCBD 752E 7B08 7241
Red Hat Inc.                               http://www.redhat.com

[Adam Williamson]

On Mon, 2018-03-19 at 10:58 +0100, Stanislav Ochotnicky wrote:
>  On Mon 19 Mar 2018 09:38:58 AM GMT Fabio Valentini wrote:
>  > So you think having to send a request to a web service instead of just
>  > parsing a string locally with one line of code is a good trade-off for
>  > allowing dashes?
>  
>  This has been mentioned several times in this thread and I think there's
>  a misunderstanding around this.
>  
>  So: When your tool/whatever works with modules it will have to have
>  module metadata available in some form. 
What if I don't want to work with modules at all, just with information
about modules?

What if I just want to write a tool that parses fedmsgs about module
builds in Koji and does something that depends on module names, streams
and versions? (e.g. some sort of changelog thing)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Nils Philippsen]

On Mon, 2018-03-19 at 08:43 -0700, Adam Williamson wrote:
>  What if I just want to write a tool that parses fedmsgs about module
>  builds in Koji and does something that depends on module names,
>  streams
>  and versions? (e.g. some sort of changelog thing) 
I haven't looked at actual fedmsgs lately but they should carry this
information as separate fields, e.g. here's how a module state change
message is specified:

https://fedora-fedmsg.readthedocs.io/en/latest/topics.html#mbs-module-sta...

The `context` field is missing here but I guess that's just a case of
the docs not keeping up with the latest changes.

Nils
-- 
Nils Philippsen    "Those who would give up Essential Liberty to
Software Engineer   purchase a little Temporary Safety, deserve neither
Red Hat             Liberty nor Safety."  --  Benjamin Franklin, 1759
PGP fingerprint:    C4A8 9474 5C4C ADE3 2B8F  656D 47D8 9B65 6951 3011

[Adam Williamson]

On Mon, 2018-03-19 at 17:18 +0100, Nils Philippsen wrote:
>  On Mon, 2018-03-19 at 08:43 -0700, Adam Williamson wrote:
>  > What if I just want to write a tool that parses fedmsgs about module
>  > builds in Koji and does something that depends on module names,
>  > streams
>  > and versions? (e.g. some sort of changelog thing)
>  
>  I haven't looked at actual fedmsgs lately but they should carry this
>  information as separate fields, e.g. here's how a module state change
>  message is specified:
>  
>  https://fedora-fedmsg.readthedocs.io/en/latest/topics.html#mbs-module-sta...
>  
>  The `context` field is missing here but I guess that's just a case of
>  the docs not keeping up with the latest changes. 
Ah, that is useful. Thanks.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Jan Kaluza]

On Mon, Mar 19, 2018 at 4:43 PM, Adam Williamson <adamwill(a)fedoraproject.org
>  wrote: 
>  On Mon, 2018-03-19 at 10:58 +0100, Stanislav Ochotnicky wrote:
>  > On Mon 19 Mar 2018 09:38:58 AM GMT Fabio Valentini wrote:
>  > > So you think having to send a request to a web service instead of just
>  > > parsing a string locally with one line of code is a good trade-off for
>  > > allowing dashes?
>  >
>  > This has been mentioned several times in this thread and I think there's
>  > a misunderstanding around this.
>  >
>  > So: When your tool/whatever works with modules it will have to have
>  > module metadata available in some form.
>
>  What if I don't want to work with modules at all, just with information
>  about modules?
>
>  What if I just want to write a tool that parses fedmsgs about module
>  builds in Koji and does something that depends on module names, streams
>  and versions? (e.g. some sort of changelog thing)
>  
Your tool can use good old buildsys.build.state.change fedmsg with the well
known name/version/release fields like this:

https://apps.stg.fedoraproject.org/datagrepper/raw?topic=org.fedoraprojec...

That's staging, because it's easier for me to find the module builds there
:).

Unfortunatelly, Koji does not add "type" field there, so you cannot find
out if that's module or not. But in case your tool is doing general
changelog per Koji build, it will work quite well with modules without you
even noticing you are working with modules.

In case of tools, you can even query koji using its api and pass the
name/version/release there, instead of NVR.

If you query Koji for the module build, you will actually find whole module
metadata in "extra" part of the build.

In case you are OK with using MBS fedmsgs, you can use the fedmsgs Nils
already sent in this thread.

Regards,
Jan Kaluza

--
>  Adam Williamson
>  Fedora QA Community Monkey
>  IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
>  http://www.happyassassin.net
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>

[Michal Novotny]

The question here is why don't we actually implement the modularity
the simple way.

Let's suppose I would like to make nodejs-8 available in EPEL7.

The most effortless way would be to create subdirectory called 8 at
src.fedoraproject.org/rpms/nodejs
with the updated spec file and sources for the bumped major version of
the nodejs package.

For fedpkg to be able to upload and download sources for that
subdirectory (submodule), the attached
single-line patch is needed in python-rpkg. There are a few more
changes needed to make srpm
importing work but it should be basically almost done as well.

Koji then just needs to learn how to build DistGit repositories
containing submodules where
submodule is any subdirectory containing a spec file. When Koji
discovers all the submodules
in a repository, it can start an srpm and a consequent rpm build for
each of them.

It is a very similar approach to what find_packages from python
setuptools implements. It
recognizes subpackages based on the presence of __init__.py file. The
only difference here
is that we will be recognizing subpackages (=submodules) by the
presence of *.spec file.

Everything else should be pretty much the same otherwise. Koji builds
nodejs-6 and nodejs-8 rpms
both with el7 disttag and both of these can go through the Bodhi
update process separately.

Of course, the question is if you can actually build and run nodejs-8
in EL7 successfully but
I think that will always be a problem no matter what the implementation is.

clime

On Mon, Mar 19, 2018 at 8:08 PM, Jan Kaluza <jkaluza(a)redhat.com&gt; wrote:
>  On Mon, Mar 19, 2018 at 4:43 PM, Adam Williamson
>  <adamwill(a)fedoraproject.org&gt; wrote:
> >
> > On Mon, 2018-03-19 at 10:58 +0100, Stanislav Ochotnicky wrote:
> > > On Mon 19 Mar 2018 09:38:58 AM GMT Fabio Valentini wrote:
> > > > So you think having to send a request to a web service instead of just
> > > > parsing a string locally with one line of code is a good trade-off for
> > > > allowing dashes?
> > >
> > > This has been mentioned several times in this thread and I think there's
> > > a misunderstanding around this.
> > >
> > > So: When your tool/whatever works with modules it will have to have
> > > module metadata available in some form.
> >
> > What if I don't want to work with modules at all, just with information
> > about modules?
> >
> > What if I just want to write a tool that parses fedmsgs about module
> > builds in Koji and does something that depends on module names, streams
> > and versions? (e.g. some sort of changelog thing)
>
>
>  Your tool can use good old buildsys.build.state.change fedmsg with the well
>  known name/version/release fields like this:
>
>
> https://apps.stg.fedoraproject.org/datagrepper/raw?topic=org.fedoraprojec...
>
>  That's staging, because it's easier for me to find the module builds there
>  :).
>
>  Unfortunatelly, Koji does not add "type" field there, so you cannot find out
>  if that's module or not. But in case your tool is doing general changelog
>  per Koji build, it will work quite well with modules without you even
>  noticing you are working with modules.
>
>  In case of tools, you can even query koji using its api and pass the
>  name/version/release there, instead of NVR.
>
>  If you query Koji for the module build, you will actually find whole module
>  metadata in "extra" part of the build.
>
>  In case you are OK with using MBS fedmsgs, you can use the fedmsgs Nils
>  already sent in this thread.
>
>  Regards,
>  Jan Kaluza
>
> > --
> > Adam Williamson
> > Fedora QA Community Monkey
> > IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
> > http://www.happyassassin.net
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>
>
>
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>

[Jan Kaluza]

On Tue, Mar 20, 2018 at 10:06 AM, Michal Novotny <clime(a)redhat.com&gt; wrote:

>  The question here is why don't we actually implement the
> modularity
>  the simple way.
>
>  Let's suppose I would like to make nodejs-8 available in EPEL7.
>
>  The most effortless way would be to create subdirectory called 8 at
>  src.fedoraproject.org/rpms/nodejs
>  with the updated spec file and sources for the bumped major version of
>  the nodejs package.
>  
What if your module contains packages coming from multiple SRPMs and they
depend on each other?

Jan Kaluza


>  For fedpkg to be able to upload and download sources for that
>  subdirectory (submodule), the attached
>  single-line patch is needed in python-rpkg. There are a few more
>  changes needed to make srpm
>  importing work but it should be basically almost done as well.
>
>  Koji then just needs to learn how to build DistGit repositories
>  containing submodules where
>  submodule is any subdirectory containing a spec file. When Koji
>  discovers all the submodules
>  in a repository, it can start an srpm and a consequent rpm build for
>  each of them.
>
>  It is a very similar approach to what find_packages from python
>  setuptools implements. It
>  recognizes subpackages based on the presence of __init__.py file. The
>  only difference here
>  is that we will be recognizing subpackages (=submodules) by the
>  presence of *.spec file.
>
>  Everything else should be pretty much the same otherwise. Koji builds
>  nodejs-6 and nodejs-8 rpms
>  both with el7 disttag and both of these can go through the Bodhi
>  update process separately.
>
>  Of course, the question is if you can actually build and run nodejs-8
>  in EL7 successfully but
>  I think that will always be a problem no matter what the implementation is.
>
>  clime
>
>  On Mon, Mar 19, 2018 at 8:08 PM, Jan Kaluza <jkaluza(a)redhat.com&gt; wrote:
>  >
>  >
>  > On Mon, Mar 19, 2018 at 4:43 PM, Adam Williamson
>  > <adamwill(a)fedoraproject.org&gt; wrote:
>  >>
>  >> On Mon, 2018-03-19 at 10:58 +0100, Stanislav Ochotnicky wrote:
>  >> > On Mon 19 Mar 2018 09:38:58 AM GMT Fabio Valentini wrote:
>  >> > > So you think having to send a request to a web service instead of
>  just
>  >> > > parsing a string locally with one line of code is a good trade-off
>  for
>  >> > > allowing dashes?
>  >> >
>  >> > This has been mentioned several times in this thread and I think
>  there's
>  >> > a misunderstanding around this.
>  >> >
>  >> > So: When your tool/whatever works with modules it will have to have
>  >> > module metadata available in some form.
>  >>
>  >> What if I don't want to work with modules at all, just with information
>  >> about modules?
>  >>
>  >> What if I just want to write a tool that parses fedmsgs about module
>  >> builds in Koji and does something that depends on module names, streams
>  >> and versions? (e.g. some sort of changelog thing)
>  >
>  >
>  > Your tool can use good old buildsys.build.state.change fedmsg with the
>  well
>  > known name/version/release fields like this:
>  >
>  > https://apps.stg.fedoraproject.org/datagrepper/
>  raw?topic=org.fedoraproject.stg.buildsys.build.state.change
>  >
>  > That's staging, because it's easier for me to find the module builds
>  there
>  > :).
>  >
>  > Unfortunatelly, Koji does not add "type" field there, so you cannot find
>  out
>  > if that's module or not. But in case your tool is doing general changelog
>  > per Koji build, it will work quite well with modules without you even
>  > noticing you are working with modules.
>  >
>  > In case of tools, you can even query koji using its api and pass the
>  > name/version/release there, instead of NVR.
>  >
>  > If you query Koji for the module build, you will actually find whole
>  module
>  > metadata in "extra" part of the build.
>  >
>  > In case you are OK with using MBS fedmsgs, you can use the fedmsgs Nils
>  > already sent in this thread.
>  >
>  > Regards,
>  > Jan Kaluza
>  >
>  >> --
>  >> Adam Williamson
>  >> Fedora QA Community Monkey
>  >> IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
>  >> http://www.happyassassin.net
>  >> _______________________________________________
>  >> devel mailing list -- devel(a)lists.fedoraproject.org
>  >> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  >
>  >
>  >
>  > _______________________________________________
>  > devel mailing list -- devel(a)lists.fedoraproject.org
>  > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  >
>
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>
>

[Michal Novotny]

On Tue, Mar 20, 2018 at 3:31 PM, Jan Kaluza <jkaluza(a)redhat.com&gt; wrote:
>  On Tue, Mar 20, 2018 at 10:06 AM, Michal Novotny
> <clime(a)redhat.com&gt; wrote:
> >
> > The question here is why don't we actually implement the modularity
> > the simple way.
> >
> > Let's suppose I would like to make nodejs-8 available in EPEL7.
> >
> > The most effortless way would be to create subdirectory called 8 at
> > src.fedoraproject.org/rpms/nodejs
> > with the updated spec file and sources for the bumped major version of
> > the nodejs package.
>
>
>  What if your module contains packages coming from multiple SRPMs and they
>  depend on each other? 
Well, my module wouldn't be coming from multiple SRPMs because my
module is just a normal package.

The problem with modularity is that the name does not really fit what
it does, which is kind of important because wrong names make
implementation much harder as you cannot rely on your common
sense.

When I think of a module, rpm ultimately comes to mind. That's kind of
a unit that makes up an operating system.

I highly doubt modularity can change it unless it invents its own package
format. So I think it would be good to call it differently even in the tooling.

So let's try to figure out a better name for modulemd.

Basically, the final product of modulemd is a repo with some built rpm
packages in it (if I saw the latest development correctly, we were squashing
multiple of those repos into a single one but we wouldn't need to do that).

For users, a repo represents something like a stream of content. So
streammd would be already a better name.

But even a better name would be coprmd because you can think about
the current "modulemd" as a copr (community project) being formally
described in a file. Product of a copr project is also a repo so it fits.

Stream name here:

https://github.com/fedora-modularity/libmodulemd/blob/master/spec.v2.yaml...

is basically a project name.


The content specified here:

https://github.com/fedora-modularity/libmodulemd/blob/master/spec.v2.yaml...

are basically definitions of SCM packages.

And then there are some fields like:

version: 20160927144203

context: c0ffee43

which doesn't really belong there imho but instead belong to the final
built repository (= copr snapshot) as a part of repodata (if anywhere).

So I think it would be good to keep talking about packages when
we talk about modules.

clime

>
> Jan Kaluza
>
>>
>>
>> For fedpkg to be able to upload and download sources for that
>> subdirectory (submodule), the attached
>> single-line patch is needed in python-rpkg. There are a few more
>> changes needed to make srpm
>> importing work but it should be basically almost done as well.
>>
>> Koji then just needs to learn how to build DistGit repositories
>> containing submodules where
>> submodule is any subdirectory containing a spec file. When Koji
>> discovers all the submodules
>> in a repository, it can start an srpm and a consequent rpm build for
>> each of them.
>>
>> It is a very similar approach to what find_packages from python
>> setuptools implements. It
>> recognizes subpackages based on the presence of __init__.py file. The
>> only difference here
>> is that we will be recognizing subpackages (=submodules) by the
>> presence of *.spec file.
>>
>> Everything else should be pretty much the same otherwise. Koji builds
>> nodejs-6 and nodejs-8 rpms
>> both with el7 disttag and both of these can go through the Bodhi
>> update process separately.
>>
>> Of course, the question is if you can actually build and run nodejs-8
>> in EL7 successfully but
>> I think that will always be a problem no matter what the implementation
>> is.
>>
>> clime
>>
>> On Mon, Mar 19, 2018 at 8:08 PM, Jan Kaluza <jkaluza(a)redhat.com&gt; wrote:
>> >
>> >
>> > On Mon, Mar 19, 2018 at 4:43 PM, Adam Williamson
>> > <adamwill(a)fedoraproject.org&gt; wrote:
>> >>
>> >> On Mon, 2018-03-19 at 10:58 +0100, Stanislav Ochotnicky wrote:
>> >> > On Mon 19 Mar 2018 09:38:58 AM GMT Fabio Valentini wrote:
>> >> > > So you think having to send a request to a web service instead
of
>> >> > > just
>> >> > > parsing a string locally with one line of code is a good
trade-off
>> >> > > for
>> >> > > allowing dashes?
>> >> >
>> >> > This has been mentioned several times in this thread and I think
>> >> > there's
>> >> > a misunderstanding around this.
>> >> >
>> >> > So: When your tool/whatever works with modules it will have to
have
>> >> > module metadata available in some form.
>> >>
>> >> What if I don't want to work with modules at all, just with
information
>> >> about modules?
>> >>
>> >> What if I just want to write a tool that parses fedmsgs about module
>> >> builds in Koji and does something that depends on module names, streams
>> >> and versions? (e.g. some sort of changelog thing)
>> >
>> >
>> > Your tool can use good old buildsys.build.state.change fedmsg with the
>> > well
>> > known name/version/release fields like this:
>> >
>> >
>> >
https://apps.stg.fedoraproject.org/datagrepper/raw?topic=org.fedoraprojec...
>> >
>> > That's staging, because it's easier for me to find the module
builds
>> > there
>> > :).
>> >
>> > Unfortunatelly, Koji does not add "type" field there, so you
cannot find
>> > out
>> > if that's module or not. But in case your tool is doing general
>> > changelog
>> > per Koji build, it will work quite well with modules without you even
>> > noticing you are working with modules.
>> >
>> > In case of tools, you can even query koji using its api and pass the
>> > name/version/release there, instead of NVR.
>> >
>> > If you query Koji for the module build, you will actually find whole
>> > module
>> > metadata in "extra" part of the build.
>> >
>> > In case you are OK with using MBS fedmsgs, you can use the fedmsgs Nils
>> > already sent in this thread.
>> >
>> > Regards,
>> > Jan Kaluza
>> >
>> >> --
>> >> Adam Williamson
>> >> Fedora QA Community Monkey
>> >> IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
>> >> http://www.happyassassin.net
>> >> _______________________________________________
>> >> devel mailing list -- devel(a)lists.fedoraproject.org
>> >> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>> >
>> >
>> >
>> > _______________________________________________
>> > devel mailing list -- devel(a)lists.fedoraproject.org
>> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>> >
>>
>> _______________________________________________
>> devel mailing list -- devel(a)lists.fedoraproject.org
>> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>>
>
>
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>

[Michal Novotny]

On Fri, Mar 23, 2018 at 7:24 AM, Michal Novotny <clime(a)redhat.com&gt; wrote:
>  On Tue, Mar 20, 2018 at 3:31 PM, Jan Kaluza
> <jkaluza(a)redhat.com&gt; wrote:
> > On Tue, Mar 20, 2018 at 10:06 AM, Michal Novotny <clime(a)redhat.com&gt; wrote:
> >>
> >> The question here is why don't we actually implement the modularity
> >> the simple way.
> >>
> >> Let's suppose I would like to make nodejs-8 available in EPEL7.
> >>
> >> The most effortless way would be to create subdirectory called 8 at
> >> src.fedoraproject.org/rpms/nodejs
> >> with the updated spec file and sources for the bumped major version of
> >> the nodejs package.
> >
> >
> > What if your module contains packages coming from multiple SRPMs and they
> > depend on each other?
>
>  Well, my module wouldn't be coming from multiple SRPMs because my
>  module is just a normal package.
>
>  The problem with modularity is that the name does not really fit what
>  it does, which is kind of important because wrong names make
>  implementation much harder as you cannot rely on your common
>  sense.
>
>  When I think of a module, rpm ultimately comes to mind. That's kind of
>  a unit that makes up an operating system.
>
>  I highly doubt modularity can change it unless it invents its own package
>  format. So I think it would be good to call it differently even in the tooling.
>
>  So let's try to figure out a better name for modulemd.
>
>  Basically, the final product of modulemd is a repo with some built rpm
>  packages in it (if I saw the latest development correctly, we were squashing
>  multiple of those repos into a single one but we wouldn't need to do that).
>
>  For users, a repo represents something like a stream of content. So
>  streammd would be already a better name.
>
>  But even a better name would be coprmd because you can think about
>  the current "modulemd" as a copr (community project) being formally
>  described in a file. Product of a copr project is also a repo so it fits.
>
>  Stream name here:
>
>  https://github.com/fedora-modularity/libmodulemd/blob/master/spec.v2.yaml...
>
>  is basically a project name.
>
>
>  The content specified here:
>
>  https://github.com/fedora-modularity/libmodulemd/blob/master/spec.v2.yaml...
>
>  are basically definitions of SCM packages. 
That link should have pointed here:
https://github.com/fedora-modularity/libmodulemd/blob/master/spec.v2.yaml...

>
> And then there are some fields like:
>
> version: 20160927144203
>
> context: c0ffee43
>
> which doesn't really belong there imho but instead belong to the final
> built repository (= copr snapshot) as a part of repodata (if anywhere).
>
> So I think it would be good to keep talking about packages when
> we talk about modules.
>
> clime
>
>>
>> Jan Kaluza
>>
>>>
>>>
>>> For fedpkg to be able to upload and download sources for that
>>> subdirectory (submodule), the attached
>>> single-line patch is needed in python-rpkg. There are a few more
>>> changes needed to make srpm
>>> importing work but it should be basically almost done as well.
>>>
>>> Koji then just needs to learn how to build DistGit repositories
>>> containing submodules where
>>> submodule is any subdirectory containing a spec file. When Koji
>>> discovers all the submodules
>>> in a repository, it can start an srpm and a consequent rpm build for
>>> each of them.
>>>
>>> It is a very similar approach to what find_packages from python
>>> setuptools implements. It
>>> recognizes subpackages based on the presence of __init__.py file. The
>>> only difference here
>>> is that we will be recognizing subpackages (=submodules) by the
>>> presence of *.spec file.
>>>
>>> Everything else should be pretty much the same otherwise. Koji builds
>>> nodejs-6 and nodejs-8 rpms
>>> both with el7 disttag and both of these can go through the Bodhi
>>> update process separately.
>>>
>>> Of course, the question is if you can actually build and run nodejs-8
>>> in EL7 successfully but
>>> I think that will always be a problem no matter what the implementation
>>> is.
>>>
>>> clime
>>>
>>> On Mon, Mar 19, 2018 at 8:08 PM, Jan Kaluza <jkaluza(a)redhat.com&gt;
wrote:
>>> >
>>> >
>>> > On Mon, Mar 19, 2018 at 4:43 PM, Adam Williamson
>>> > <adamwill(a)fedoraproject.org&gt; wrote:
>>> >>
>>> >> On Mon, 2018-03-19 at 10:58 +0100, Stanislav Ochotnicky wrote:
>>> >> > On Mon 19 Mar 2018 09:38:58 AM GMT Fabio Valentini wrote:
>>> >> > > So you think having to send a request to a web service
instead of
>>> >> > > just
>>> >> > > parsing a string locally with one line of code is a good
trade-off
>>> >> > > for
>>> >> > > allowing dashes?
>>> >> >
>>> >> > This has been mentioned several times in this thread and I
think
>>> >> > there's
>>> >> > a misunderstanding around this.
>>> >> >
>>> >> > So: When your tool/whatever works with modules it will have to
have
>>> >> > module metadata available in some form.
>>> >>
>>> >> What if I don't want to work with modules at all, just with
information
>>> >> about modules?
>>> >>
>>> >> What if I just want to write a tool that parses fedmsgs about
module
>>> >> builds in Koji and does something that depends on module names,
streams
>>> >> and versions? (e.g. some sort of changelog thing)
>>> >
>>> >
>>> > Your tool can use good old buildsys.build.state.change fedmsg with the
>>> > well
>>> > known name/version/release fields like this:
>>> >
>>> >
>>> >
https://apps.stg.fedoraproject.org/datagrepper/raw?topic=org.fedoraprojec...
>>> >
>>> > That's staging, because it's easier for me to find the module
builds
>>> > there
>>> > :).
>>> >
>>> > Unfortunatelly, Koji does not add "type" field there, so you
cannot find
>>> > out
>>> > if that's module or not. But in case your tool is doing general
>>> > changelog
>>> > per Koji build, it will work quite well with modules without you even
>>> > noticing you are working with modules.
>>> >
>>> > In case of tools, you can even query koji using its api and pass the
>>> > name/version/release there, instead of NVR.
>>> >
>>> > If you query Koji for the module build, you will actually find whole
>>> > module
>>> > metadata in "extra" part of the build.
>>> >
>>> > In case you are OK with using MBS fedmsgs, you can use the fedmsgs Nils
>>> > already sent in this thread.
>>> >
>>> > Regards,
>>> > Jan Kaluza
>>> >
>>> >> --
>>> >> Adam Williamson
>>> >> Fedora QA Community Monkey
>>> >> IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin .
net
>>> >> http://www.happyassassin.net
>>> >> _______________________________________________
>>> >> devel mailing list -- devel(a)lists.fedoraproject.org
>>> >> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > devel mailing list -- devel(a)lists.fedoraproject.org
>>> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>>> >
>>>
>>> _______________________________________________
>>> devel mailing list -- devel(a)lists.fedoraproject.org
>>> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>>>
>>
>>
>> _______________________________________________
>> devel mailing list -- devel(a)lists.fedoraproject.org
>> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>>

[Kamil Paral]

On Mon, Mar 19, 2018 at 8:08 PM, Jan Kaluza <jkaluza(a)redhat.com&gt; wrote:

>  Your tool can use good old buildsys.build.state.change fedmsg with
> the
>  well known name/version/release fields like this:
>
>  https://apps.stg.fedoraproject.org/datagrepper/
>  raw?topic=org.fedoraproject.stg.buildsys.build.state.change
>  
Initially I was very concerned about the idea of having to query Koji for
each module to get NSVC, because in Taskotron we run ten thousand tasks per
day and every unnecessary network request means slower execution, higher
risk of failure and more logs to debug. We try to minimize the number of
network requests as much as we can. However, if the structured metadata
is/will be available in fedmsgs, I think I don't have a use case at hand
that I could use to object to this. I still think it's a bad idea, though,
and will bite us in the long run.


>  That's staging, because it's easier for me to find the module builds there
>  :).
>
>  Unfortunatelly, Koji does not add "type" field there, so you cannot find
>  out if that's module or not.
>  
Is that something that will get resolved soon? Because it seems like a big
blocker to any automated processing. If we want to run some tasks just on
rpms and some tasks just on modules, how do we distinguish that?

[Jan Kaluza]

On Tue, Mar 20, 2018 at 3:18 PM, Kamil Paral <kparal(a)redhat.com&gt; wrote:

>  On Mon, Mar 19, 2018 at 8:08 PM, Jan Kaluza
> <jkaluza(a)redhat.com&gt; wrote:
>
> > Your tool can use good old buildsys.build.state.change fedmsg with the
> > well known name/version/release fields like this:
> >
> > https://apps.stg.fedoraproject.org/datagrepper/raw?topic=
> > org.fedoraproject.stg.buildsys.build.state.change
> >
>
>  Initially I was very concerned about the idea of having to query Koji for
>  each module to get NSVC, because in Taskotron we run ten thousand tasks per
>  day and every unnecessary network request means slower execution, higher
>  risk of failure and more logs to debug. We try to minimize the number of
>  network requests as much as we can. However, if the structured metadata
>  is/will be available in fedmsgs, I think I don't have a use case at hand
>  that I could use to object to this. I still think it's a bad idea, though,
>  and will bite us in the long run.
>
>
> >
> > That's staging, because it's easier for me to find the module builds
> > there :).
> >
> > Unfortunatelly, Koji does not add "type" field there, so you cannot find
> > out if that's module or not.
> >
>
>  Is that something that will get resolved soon? Because it seems like a big
>  blocker to any automated processing. If we want to run some tasks just on
>  rpms and some tasks just on modules, how do we distinguish that?
>  
You can query the Koji for build id and check the type of a build.

I don't think that feature request is even reported to Koji upstream,
because nobody needed that and I found that yesterday.

It is however not new situation. There is the same "issue" for
"images". If
you want to run some tasks just for RPMs and not for images, you have to do
the filtering like this anyway. Therefore I don't really consider this as
real blocker which needs fix urgently.


>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>
>

[Adam Williamson]

On Mon, 2018-03-19 at 09:27 +0100, Petr Šabata wrote:
>  This, AFAIK, only affects what you see in some parts of the
>  build and update system (and on the bus). 
That's a very important parenthesis, as we as a project do a lot of
work with fedmsg messages.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Owen Taylor]

On Fri, Mar 16, 2018 at 4:04 PM, Dennis Gregorovic <dgregor(a)redhat.com&gt;
wrote:

>  The challenge here is that those module builds are squeezed into a
> N-V-R
>  format when imported into Koji.  The N-V-R format is used for all build
>  types in Koji and it works well for its purpose, but it's also not
>  realistic to expect that all content types are going to natively use N-V-R
>  outside of Koji. My suggestion is that we consider the N-V-R format of
>  modules to be a representation internal to koji and that N:S:V:C is the
>  format used to represent modules to users.  If that means tweaking the Koji
>  UI we can look into that.
>  
There's also a terminology problem that gets less confusing with the :
separator. When compared to

 Name-VERSION-Release.DistTag

it would be fairly confusing to have:

 Name-Stream-VERSION.Context

while:

  Name:Stream:VERSION:Context

is slightly less confusing. of course, it might have been better to call
'Version' in a module version Build or something, but I think that ship has
sailed.

In the end, modules are not packages, and it's necessarily required or good
if they are named in the same format. I'd also like to see Koji's
representation of a module version stay inside Koji as much as possible.

Owen

[Adam Williamson]

On Fri, 2018-03-16 at 16:04 -0400, Dennis Gregorovic wrote:
>  modules are not RPMs.  I would not expect them to necessarily use the
> same
>  format as RPMs.  If we take koji out of the equation, 
That's an extremely big and invalid 'if', though.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Dennis Gilmore]

El vie, 16-03-2018 a las 15:19 -0700, Adam Williamson escribió:
>  On Fri, 2018-03-16 at 16:04 -0400, Dennis Gregorovic wrote:
>  > modules are not RPMs.  I would not expect them to necessarily use
>  > the same
>  > format as RPMs.  If we take koji out of the equation,
>  
>  That's an extremely big and invalid 'if', though. 
koji is mirroring
rpm's behaviour, the if would have to be if you took
rpm out of the equation.

Dennis

[Kevin Kofler]

Randy Barlow wrote:
>  It sounds like this breaks a great deal of tooling. Can we
> reconsider
>  switching away from - separators in modules? Is allowing streams to have
>  -'s in them important enough to break so many tools? 
+1, I think allowing this ambiguity is a very bad idea.

        Kevin Kofler

[Adam Williamson]

On Thu, 2018-03-15 at 16:14 +0100, Patrick Uiterwijk wrote:
>  One of the interesting things they wanted to also allow: dashes in
> streams.
>  As a consequence, when you get an N-S-V.C as modules are represented
>  in Koji builds, doing a .rsplit('-', 2) will not give you Name,
>  Stream, Version.Context per se.
>  You could totally have a module called
>  nodejs-my-stream-5-20170314.abcd, with name=nodejs,
>  stream=my-stream-5, version=20170314, context=abcd.
>  There is no way for you to independently figure out what the NSVC
>  components are, you will need to ask Koji, and use its name, version
>  and release fields (with name=name, version=stream,
>  release=version(.context)). 
...

>  I hope that this is useful information for anyone else finding
>  themselves having to parse NVRs/NSVs. 
1. Yes, it helps, thanks.
2. This is goddamn terrible and whoever is responsible should be
ashamed.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net